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Appendix A. About the study 


This appendix describes the impetus and rationale for the study of how states are assessing schools’ 
implementation of multi-tiered system of supports (MTSS)/response to intervention (RTI) frameworks and reviews 
selected literature on key MTSS/RTI practices. Driven by the 2004 reauthorization of the Individuals with 
Disabilities Education Act (IDEA, 2004), MTSS/RTI is a framework implemented in states to identify and support 
students with learning disabilities. 


Why this study? 


According to the Tennessee Department of Education, too many Tennessee elementary-school-age students read 
below grade-level standards. Only 43 percent of grade 3 students scored proficient in reading on the 2015 
Tennessee state assessment. Furthermore, grade 3 students from historically underserved groups were less likely 
to be proficient in reading: 32 percent of economically disadvantaged students; 30 percent of Black, Hispanic, or 
American Indian/Alaskan Native students; 22 percent of English learner students; and 20 percent of students with 
disabilities (Tennessee Department of Education, 2016). These results are unsatisfactory to state policymakers 
and concerning because students with limited reading ability are at risk for grade retention and dropout. Students 
who struggle with reading in early elementary school are four times more likely than other students to leave 
school before graduation (Cannon & Lipscomb, 2011; Hernandez, 2011). 


Tennessee’s goal is to have at least 75 percent of grade 3 students reading proficiently by 2025 (Tennessee 
Department of Education, 2016). State leadership is addressing this challenge by establishing initiatives, such as 
the Response to Instruction and Intervention (RTI*) framework, that underscore the importance of intervening 
early to prevent reading problems from escalating. The RTI* framework seeks to provide high-quality, rigorous, 
and differentiated instruction for all students through tier 1 supports and uses universal screening and data-based 
decisionmaking processes and tools to identify when students may need more targeted intervention and supports 
through tiers 2 and 3 (Tennessee Department of Education, 2017a). This framework enables educators to 
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intervene early for students with persistent academic difficulties and to identify students who may have a specific 
learning disability (Tennessee Department of Education, 2017a). 


State leaders recognized that they could improve RTI* implementation, and in January 2017 the Tennessee 
Department of Education (TDOE) published Response to Instruction and Intervention Framework, a manual that 
provided updated guidance to districts and schools on how to implement RTI practices (Tennessee Department of 
Education, 2017b). In needs-sensing conversations, state leaders also acknowledged the challenges created by a 
lack of high-quality, accurate data for making decisions about how to support all Tennessee schools in 
implementing RTI? (N. Schwartz, personal communication, February 27, 2017). State leaders wanted to know how 
schools are implementing RTI? practices in order to understand whether the practices were improving literacy 
outcomes. To inform its examination of how Tennessee schools are implementing RTI?, TDOE leaders were 
interested in learning how other states support schools by using tools to assess implementation fidelity and 
progress. 


Although the Regional Educational Laboratory (REL) Appalachia’s research partnership with TDOE focuses 
primarily on understanding how RTI can improve early literacy outcomes, TDOE leaders also expressed a broader 
interest in learning about tools that cover other content areas (such as math or behavior) and grade levels (middle 
and high school) as a result of recent RTI policy and practice shifts (Z. Stone, personal communication, July 12, 
2017). As reflected in the most recent reauthorization of the Elementary and Secondary Education Act, under the 
Every Student Succeeds Act (2015), states including Tennessee are moving away from using an RTI framework for 
academics and a different framework for behavior and instead are using an integrated system of supports referred 
to as a multi-tiered system of supports (MTSS; Tennessee Department of Education, 2018; Zumeta Edmonds, 
2016). 


RTI is more closely associated with special education and data-driven decisionmaking to identify students with 
learning disabilities, whereas MTSS has a greater focus on general education and integrates both academics and 
behavior (Burns, Jimerson, VanDerHeyden, & Deno, 2016). Both RTI and MTSS, however, use a data-based 
framework, include processes and measures for screening all students, provide instruction to all (often called 
tier 1) and tiered intervention supports to struggling students (often called tiers 2 and 3), use evidence-based 
practices, and monitor students’ progress (Burns et al., 2016). The terms RTI and MTSS are sometimes used 
interchangeably (National Center on Response to Intervention, 2015, n.d.). To conduct a comprehensive review 
of assessment tools in use, the REL Appalachia identified tools assessing RTI implementation that address RTI and 
MISS,’ all grade levels (elementary, middle, and high school), and multiple content areas (for example, literacy 
and math). 


This study is intended to serve state leaders who are interested in developing or adapting a tool to assess MTSS/RTI 
implementation. Therefore, the study does not include tools that states adopted without modification from other 
organizations or states. By showing how other states are assessing implementation of MTSS/RTI practices, this 
descriptive study will help TDOE, as well as regional educational laboratories and other states, develop a strategy 
to learn more about how MTSS/RTI is being implemented in schools. Findings from the study may assist states 
without an MTSS/RTI implementation assessment tool to adopt, modify, or develop one that best meets their 
needs. In addition, states with an implementation assessment tool may use the study findings to review and reflect 
on the tool in order to affirm or refine it. The study findings also provide examples of approaches that state 
personnel used to roll out the tools and train school personnel in using tools and interpreting findings. 


1 To promote readability and alignment with TDOE’s current language on RTI and to respond to guidance from REL Appalachia’s study 
advisory panel of RTI experts, this report uses the terms MTSS/RTI. An exception is when describing the website and document review, 
when the report uses the term that accurately represents the framework that a particular state uses. 
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Examining the fidelity of implementation of the core components of an MTSS/RTI framework is the first critical 
step to understanding whether MTSS/RTI leads to better student outcomes. The development and use of better 
tools for assessing implementation fidelity may provide a catalyst for states to evaluate and assess the impact of 
MTSS/RTI. 


Summary of multi-tiered system of supports/response to intervention key practices 


Below is a summary of the research on key MTSS/RTI practices as they pertain to reading and math, with selected 
references. It is beyond the scope of this study to conduct a systematic literature review for each practice. Because 
the current study focused on MTSS/RTI implementation assessment tools specific to academics, particularly 
reading and math, this summary excludes literature on practices pertaining only to behavior within an MTSS 
framework. 


Administer assessments 


Screen all students, and target intervention support. Early identification of students at risk for long-term difficulties 
in reading and math begins with systematic screening using validated measures and processes. Screening takes 
place near the beginning of the school year and at least once in the middle of the year for all students in 
elementary school (Gersten et al., 2008; Gersten et al., 2009).* Screening data can identify students in need of 
intervention support and can be aggregated to examine the adequacy of the core curriculum and the effectiveness 
of different instructional strategies used in a school. 


Monitor struggling students’ responsiveness to intervention. A signature feature of MTSS/RTI is frequent 
assessment of student performance on a valid and reliable progress monitoring measure (for example, oral 
reading fluency in the early grades, Deno, 2016; Jenkins, Hudson, & Johnson, 2007; number identification in early 
math, Foegen, Jiban, & Deno, 2007). Examined over time, these measures depict students’ growth and signal 
whether their response to intervention is on track to reach a learning goal within a specified period of time 
(Foegen & Deno, 2001; Fuchs, Deno, & Mirkin, 1984; Fuchs, Fuchs, Hamlett, & Stecker, 1991). Such progress 
monitoring, in addition to periodic universal screening, is seen as a way to reduce the number of students falsely 
identified as needing additional support (Compton et al., 2010) or to increase the likelihood that students referred 
for special education evaluations qualify for services (VanDerHeyden, Witt, & Gilbertson, 2007). 


For reading, the recommended practice in elementary school is to monitor the progress of students in tier 2 at 
least monthly and of students in tier 3 at least weekly to determine their response to intervention (Gersten et al., 
2008).? For math the recommended practice is to monitor the progress of students in tier 2 and tier 3 using 
outcome measures at least monthly and curriculum-based measures daily to biweekly to determine their response 
to intervention (Gersten et al., 2009). 


To meet the standards for educational and psychological testing, screening and progress monitoring measures 
must provide evidence of validity and reliability (American Educational Research Association, 2014). 


Offer multiple tiers of instruction and intervention 


High-quality classroom-based instruction, often called tier 1. In tier 1, all students are provided high-quality core 
instruction using evidence-based programs (Denton, Vaughn, & Fletcher, 2003; Stoiber & Gettinger, 2016) and 
aligned with state standards (Jimerson, Stein, Haddock, & Shahroozi, 2016). Evidence-based programs are defined 
as practices that integrate the best available research with clinical expertise in the context of student 


2 Gersten et al. (2008) classified the level of evidence for this practice as moderate, because the evidence was supported by a series of high- 
quality correlational studies with replicated findings (Compton, Fuchs, Fuchs, & Bryant, 2006; Fuchs, Fuchs, & Compton, 2004; McCardle, 
Scarborough, & Catts, 2001; O’Connor & Jenkins, 1999; Scarborough, 1998). 

3 Gersten et al. (2008) classified the level of evidence for this practice as low for tier 2 students on the basis of consensus of the panel. 
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characteristics, culture, and preferences (Stoiber & Gettinger, 2016). All students receive reading and math core 
instruction during a designated block of time, which includes whole-class and small-group instruction. Small-group 
instruction enables the teacher to differentiate instruction based on assessments of students’ current level and 
skills (Gersten et al., 2008; Gersten et al., 2009).* In tier 1, students are assembled for differentiated small-group 
instruction with a focus on one or more skills relevant to the core components of reading and math (Gersten et 
al., 2008; Gersten et al., 2009). According to the National Reading Panel (2000), research does not provide a clear, 
comprehensive model of how to teach reading to students in elementary school, but the panel did identify five 
essential components of reading instruction: phonemic awareness, word attack, fluency, vocabulary, and 
comprehension. For math, areas of emphasis within each grade from prekindergarten through grade 8 are 
outlined in the National Council of Teachers of Mathematics’ Curriculum Focal Points (2006) guidance document. 


Increasingly intensive tiers of intervention are often called tiers 2 and 3. Students who are identified by a screening 
assessment as performing below grade level receive an intervention. Instructional practices to increase 
instructional intensity include varying the amount of instructional time, group size, frequency of progress 
monitoring for determining student progress as a result of the intervention, and the teacher’s role in providing 
the instruction (Balu et al., 2015; Elbaum, Vaughn, Hughes, & Moody, 2000; Mellard, McKnight, & Jordan, 2010; 
Sugai & Horner, 2009; Wanzek & Vaughn, 2007). 


In the case of literacy, tier 2 interventions include providing “intensive, systematic instruction on up to three 
foundational reading skills in small groups to students who score below the benchmark on universal screening. 
Typically, these groups meet between three and five times a week, for 20 to 40 minutes” (Gersten et al., 2008, 
p. 19).° For math, tier 2 interventions are “aimed at building targeted mathematics proficiencies” and are 
“typically provided for 20 to 40 minutes, four to five times each week” (Gersten et al., 2009, p. 5). 


The multiple tiers of instruction approach also supports interventions for students identified as performing “far 
below” grade level on reading and math. There is some evidence that providing students with two tiers of math 
instruction (through validated classroom instruction supplemented with high-quality tutoring) compared with just 
one tier of math instruction has positive student learning outcomes (Fuchs et al., 2008). Research evidence and 
expert judgment recommend providing “intensive instruction on a daily basis that promotes the development of 
the various components of reading proficiency to students who show minimal progress after reasonable time in 
tier 2 small group instruction (tier 3)” (Gersten et al., 2008, p. 26). Some experts, however, differ on whether these 
students should first receive tier 2 instruction or go directly to tier 3 (Vaughn & Swanson, 2015). In practice, tier 
placement may not correspond precisely with reading level as measured by screening tests (Balu et al., 2015; 
Wanzek & Vaughn, 2007). Students may cycle in and out of intervention tiers. 


Support data-based decisionmaking 


Screen all students, target intervention support, and monitor struggling students’ responsiveness to interventions. 
Using data in decisionmaking is a prominent practice in an MTSS/RTI system. Schools implementing multiple tiers 
of reading and math instruction and intervention use assessments to screen all students and target intervention 
support to those identified as below grade-level benchmarks, monitor struggling students’ responsiveness to 
interventions, and in some cases inform decisions about eligibility for special education services (Burns & 
Ysseldyke, 2005; Fletcher, Coulter, Reschly, & Vaughn, 2004; Fuchs & Fuchs, 2006; Gersten et al., 2008; Gersten 
et al., 2009; VanDerHeyden, Witt, & Barnett, 2005). 


4 Gersten et al. (2008) classified the level of evidence for this practice as low, because the evidence was supported by a correlational study 
that examined student reading growth when teachers employed a specific differentiation program (Connor et al., 2009). 

5 Gersten et al. (2008) classified the level of evidence for this practice as strong, because the evidence is based on 11 studies meeting What 
Works Clearinghouse standards or meeting standards with reservations. 
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Schools can choose from a range of commercial and free measures for monitoring progress. Elements of an 
assessment system include standardization of screening procedures; grade-level benchmarks or expectations; 
designated risk levels; ease and efficiency of administration; and documented reliability, validity, and diagnostic 
accuracy of the screening measures. 


School-based data teams, including data experts, teachers, and school leaders, should develop a formal procedure 
for making informed decisions using data collected from assessment and progress monitoring measures 
(Schildkamp, Poortman, & Handelzalts, 2016). Professional development opportunities specifically targeting 
effective data use can have positive effects on student academic achievement and should be provided for teachers 
and school leaders (Lai & McNaughton, 2016). 


Inform decisions regarding eligibility for special education services. Also, as part of an RTI framework for academic 
instruction, school staff are encouraged to intensify intervention or consider eligibility for special education 
services (for example, under the category of specific learning disability) when progress measures indicate 
insufficient response to interventions (Fletcher et al., 2004). 


Support infrastructure practices for MTSS/RTI implementation 


To implement an RTI system, schools usually form school-level teams, also called problem-solving teams, and 
allocate time for them to meet and evaluate the progress of students moving toward grade-level standards by 
using screening and progress monitoring data (Burns, Deno, & Jimerson, 2007; Schulte, 2016). Problem-solving 
teams are expected to follow an established process for using data to make decisions. General stages in the 
process include identifying the problem, defining the problem, exploring alternative solutions to the problem, 
applying a solution, and looking at the effects (Individuals with Disabilities Education Improvement Act, 2004). 
Schools should also designate support staff to guide teacher implementation of the program (Borman, Hewes, 
Overman, & Brown, 2003; Glennan, Bodilly, Galegher, & Kerr, 2004). In an RTI framework, support staff, such as 
instructional coaches, focus on working with teachers to understand their data or focus on improving reading and 
math instruction and making changes in intervention practices based on the data. 
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Appendix B. Methods 


This appendix includes a description of the sample selection for the in-depth tool analysis, the names of states 
and tools included or not included in the in-depth analysis, the interview selection criteria, the website and 
document review protocol, and the methods for coding and analyzing key multi-tiered system of supports 
(MTSS)/response to intervention (RTI) practices. 


Sample selection for in-depth analysis 


The study team used the process described below to arrive at a sample of 21 states (with 31 tools) for the in-depth 
tool analysis: 


e Step 1: Conduct a website and document search for all 50 states and the District of Columbia to find tools 
used to support MTSS/RTI implementation. 


e Step 2: Record information about MTSS/RTI systems-—level tools that states verified they were using as of May 
11, 2018, and record whether the tool was in development or under revision at that time. This step included 
tools that were developed by the state or adapted from existing tools as well as tools from other states or 
organizations that were used without being modified. 


e Step 3: Use the inclusion and exclusion criteria in box B1 to select for more in-depth analysis, arriving at 21 
states (with 31 tools). 


Box B1. Inclusion and exclusion criteria for in-depth analysis of implementation assessment tools 
Inclusion criteria 
e Tool is publicly available on a state’s website or accessible to the public on request. 


e Tool focuses on a systems-level assessment of multi-tiered system of supports (MTSS) or response to interventions (RTI). 
This refers to the comprehensiveness of the tool, in that it includes practices that cover all aspects of the system, as 
opposed to focusing on one specific component of the MTSS/RTI framework. 


e Tool was developed by the state or adapted from existing instruments, so they are tailored to the state’s need. 
Exclusion criteria 


e Tool is from another state (such as Florida’s Self-Assessment of MTSS and Michigan’s Integrated Behavior and Learning 
Support Initiative’s Reading Tiered Fidelity Inventory) and has not been modified or adapted by the state. 


e Tool is from another organization (for example, the National Center on Response to Intervention, the National Center on 
Intensive Intervention, or the National Implementation Research Network) and has not been modified or adapted by the 
state. 


e Tool focuses on a specific component of a state’s MTSS/RTI framework (for example, progress monitoring) rather than 
taking a systems-level perspective of the MTSS/RTI framework. 


e Document describes a state’s MTSS/RTI framework but is not a tool used to provide ratings or scores (for example, 
Kansas’s Innovation Configuration Matrix or Oregon’s Response to Instruction and Intervention Installation Matrix). 


e Tool focuses exclusively on behavior. 
e Tool is not publicly available or accessible (for example, requires a special logon to access). 
e = Tool is in development or under revision. 


e Tool is not specific to MTSS/RTI but can be applied to other innovations or initiatives more broadly (for example, District 
Capacity Assessment from the National Implementation Research Network and California’s Differentiated Assistance 
Protocols for Local Educational Agencies). 


e Tool is an abbreviated or abridged version of a tool that is already included in the in-depth analysis. 


e State verified not having a tool. 


Source: Authors’ decisions about tool inclusion criteria established at the beginning of the study in 2018. 
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Of the 50 states and the District of Columbia, 5 states (10 percent) were excluded because state personnel chose 
not to participate (2 states) or did not verify the data collected for the study (3 states; table B1). For example, one 
state required submission of a formal application to conduct research before verifying data collected for the state; 
the study timeframe precluded submitting a formal application. One state chose not to participate in the study 
because of concerns about potential misrepresentation of the state’s MTSS/RTI resources. The study team wanted 
state personnel to verify all data collected about the tool to ensure the accuracy of the information. Therefore, 
when a state official did not verify the data, that state was excluded from the overall study sample. 


Twenty-five states (49 percent) were included in the overall study sample but were not included in the in-depth 
tool analysis because the state reported not having a tool to support MTSS/RTI implementation (12 percent), a 
tool was in development (7 percent), a tool was under revision (6 percent), or other exclusion criteria applied 
(24 percent; for example, state was using a tool from another state or organization without modification, state 
was using a general tool that encompassed more than MTSS/RTI, or a state-developed tool was not publicly 
available; see table B1). Because this study is intended as a resource for state leaders who are interested in 
developing or adapting a tool to assess MTSS/RTI implementation, it does not include tools that states adopted 
without modification from other organizations or states. Ten states (20 percent) used tools from other 
organizations or states without modification. 


The remaining 21 states (41 percent) had a tool that the state developed or adapted to support MTSS/RTI 
implementation and were included in the in-depth tool analysis. Of these 21 states, 14 (66 percent) had one tool 
that met the inclusion criteria, 5 (24 percent) had two tools that met the inclusion criteria, 1 (5 percent) had three 
tools that met the inclusion criteria, and 1 (5 percent) had four tools that met the inclusion criteria (table B2). Two 
states had distinct tools to focus on reading and math separately, two states used more than one tool to target 
elementary and secondary grade levels separately, two states used multiple tools to assess the perspectives of 
both administrators and staff, and one state used multiple tools to assess implementation at both the district and 
school levels. 
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Table B1. Overall study sample 
Number of 


states Percentage 
Status of state multi-tiered system of supports/response to intervention tools (N= 51) of states 


States excluded from overall study sample 


State personnel chose not to participate in the study 2 4 
No state personnel were designated to answer verification questions about the tool 1 2 
State did not verify the lack of an implementation tool i 2 
State did not verify the implementation assessment tools the study team identified 1 2 
Subtotal 5 10 
States included in overall study, but not in in-depth tool analysis? 

State is using only tools from other states or organizations without modification 6 12 
State is using a tool from another state or organization without modification and a general tool? 3 6 
State is using a tool from another state or organization without modification and has another tool 1 2 
under revision? 

State is using only a general tool* 1 2 
State-developed tool is not publicly available? 1 2 
Tool is in development 4 7 
Tool is under revision 3 6 
State verified not having a tool 6 12 
Subtotal 25 49 
States included in overall study and in in-depth tool analysis 

State is using only a tool that it developed or adapted and that is publicly available or accessible 16 31 
to the public. 

State is using a tool that it developed or adapted from an existing tool and that is publicly 5 10 
available or accessible to the public, and state is using a tool from another state or organization 

without modification? 

Subtotal 21 41 


a. The key findings of the main report simplify the reporting categories for states that are included in the overall study but not in the in-depth analysis. The 
three states that are categorized as “State is using a tool from another state or organization without modification and a general tool” are counted only as 
using an existing tool without modification. The one state categorized as “State is using a tool from another state or organization without modification and 
has another tool under revision” is counted only as using an existing tool without modification. The remaining two states are described in a footnote: one 
state is using only a general tool that is not specific to multi-tiered system of supports/response to intervention, and one state-developed tool is not publicly 
available. 

b. See table B3 for the names of the tools from other organizations or states that are being used without modification. 

Source: Authors’ analysis of tools based on study inclusion criteria in 2018. 


Table B2. Study sample for in-depth analysis of multi-tiered system of supports/response to intervention 
implementation assessment tools 


Number of 
states Percentage of 
Number of tools meeting study inclusion criteria (n= 21) states 
State has one tool that met study inclusion criteria 14 66 
State has two tools that met study inclusion criteria 5 24 


State has three tools that met study inclusion criteria 
State has four tools that met study inclusion criteria 


Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 
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Names of states and tools included in the in-depth analysis 


Table B3 lists the implementation assessment tools included in the in-depth analysis, organized by state. The table 
also includes the publication year and hyperlink for each tool. 


Table B3. State and tool names for in-depth analysis of multi-tiered system of supports/response to 
intervention implementation assessment tools, with publication year and hyperlink 


Publication 

State Assessment tool name year PNET) ey fiavae) anand X-allal< 

Alabama Implementation of Response to 2009 The tool is available to the public on request. 
Instruction (Rtl) in Alabama Self- 
Assessment Tool 

Alaska District and School RTI Readiness 2009 https://education.alaska.gov/esea/rti/docs/alaska_rti 
Checklist guidance.pdf (see appendix H) 

Colorado CO-MTSS Building Leadership Team 2017 https://www.cde.state.co.us/mtss/mtss-schoolself- 
(BLT) Self-Assessment assessmentevaluationtool 
MTSS Leadership Team (MLT) Self- 2017 https://www.cde.state.co.us/mtss/mtss-mlt-self- 
Assessment assessment-jan2017 

Connecticut Scientific Research-Based Interventions 2012 http://spdg.serc.co/assets/program-evaluation/SRBI- 
(SRBI) Self-Assessment Self-Assessment-for-2012-15-Cohort.docx 

Florida Self-Assessment of MTSS 2015 http://www. floridarti.usf.edu/resources/program_eval 
Implementation (SAM Version 2.0) uation/sam/sam.pdf 

Idaho Idaho RTI Fidelity Implementation 2012 https://idahotc.com/Resources/View/ID/274 
Rubric 

Illinois Self-Assessment of MTSS 2017 http://www. rti- 
Implementation—lllinois (SAM-1) innovations.com/uploads/1/0/8/2/10825600/e. sam- 

i__version_2.0__-_1.10.pdf 
Kansas Checklist for Implementation Readiness 2017 The tool is available to the public on request. 


and the Kansas Multi-tier System of 

Supports Innovation Configuration 

Matrix (ICM) 

Process Implementation Tool and the 2017 The tool is available to the public on request. 
Kansas Multi-tier System of Supports 

Innovation Configuration Matrix (ICM) 


Integrated MTSS Implementation Scale 2017 The tool is available to the public on request. 
— Instructional Staff 
Integrated MTSS Implementation Scale 2017 The tool is available to the public on request. 
— Administrators 
Kentucky Self-Reflection Tool for Sites with 2018 https://education.ky.gov/educational/int/ksi/Docume 
Exemplary RTI Practices nts/Self-Reflection Tool RT| Model Site.doc 
Michigan Reading Tiered Fidelity Inventory (R -TFI) 2018 https://miblsi.org/sites/default/files/Documents/Evalu 
Elementary-Level Edition ation/Fidelity/RTFl/website March2018 R- 
TFI%20Elementary-Level%20Edition v1.3.pdf 
Reading Tiered Fidelity Inventory (R -TFl) 2018 https://miblsi.org/sites/default/files/Documents/Evalu 
Secondary-Level Edition ation/Fidelity/RTFI/Jan 2018 R-TFI_ Secondary- 
Level%20 Edition v1.3 SECURED.pdf 
Reading Tiered Fidelity Inventory 2017 https://miblsi.org/sites/default/files/Documents/Evalu 
Reading Screening Items for Secondary ation/Fidelity/RTFI/July%202017-R- 
Schools TFI%20Reading%20Screening%20ltems%20for%20Sec 
ondary%20Schools_v1.1.pdf 
Minnesota Minnesota Reading Tiered Fidelity 2016 The tool is available to the public on request. 
Inventory 
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State 


New York 


North Carolina 


North Dakota 


Oklahoma 
Oregon 


Pennsylvania 


Texas 


Vermont 


Wisconsin 


Wyoming 


Assessment tool name 

Minnesota Math Tiered Fidelity 
Inventory 

New York State Self-Assessment Tool for 
Rtl Readiness (targets elementary 
schools) 

NYS Self-Assessment Tool for RTI Middle 
School Readiness & Implementation 
Survey - UNABRIDGED 

North Carolina Self-Assessment of MTSS 
(SAM) 

NDMTSS Fidelity of Implementation 
Rubric 

OTISS Fidelity Assessment 

District Implementation Evaluation Tool: 
School Based form (DIET-SB2) 

Using Response to Intervention for SLD 
Determination: School Building 
Application (K-12) 


Campus Needs Assessment Tool (CNAT): 
Reading 

Campus Needs Assessment Tool (CNAT): 
Mathematics 

Vermont Multi-tiered System of 
Supports Response to Intervention and 
Instruction (MTSS-RtII) Self-Assessment 
School-wide Implementation Review 
(SIR) 

School Rtl All-Staff Perception Survey 


MTSS Implementation Checklist for 
Literacy 


Publication 


year 
2016 


2010 


2017 


2015 


2014 


2017 
2015 


2017 


2018 


2018 


2014 


2015 


2016 


2016 


PWETIEV ey fiavace) am and eX=yallali< 
The tool is available to the public on request. 


https://nysrti.org/files/documents/resources/forms/n 
ew_york_state_self_assessment_tool_ for_rti_readines 
s_as_ word doc updated 1 8 14.pdf 
https://nysrti.org/files/resources/rti_tools forms/una 
bridged _nys mtss_ self-assessment _- 


middle_school.pdf 
http://portfolio.du.edu/downloaditem/377501 


https://www.nd.gov/dpi/uploads/194/2.NDMTSSFideli 


tyoflmplementationRubric.pdf 
http://www.otiss.net/otiss-tools 


http://www.oregonrti.org/s/DIET-SB2-ORTIi-Coachs- 
Scoring-Guide-Final.docx 


https://www.pattan.net/getattachment/Multi-Tiered- 
System-of-Support/Response-to-Intervention-RTI/RTI- 
SLD- 
Determination/UsingResponselntervention_SL.PDF/?la 
ng=en-US 
http://resources.buildingrti.utexas.org/booklets/CNAT 
Reading 2018.pdf 
http://resources.buildingrti.utexas.org/booklets/CNAT 
Mathematics _2018.pdf 
http://vriuvmmtss.wpengine.com/wp- 
content/uploads/2014/06/MTSS-Rtll-SelfAssessment- 
062014.pdf 
https://www.wisconsinrticenter.org/assets/files/SIR/P 
rintFriendlySIR.pdf 
https://www.wisconsinrticenter.org/assets/files/AllSta 
ff/AllStaff%20Survey042016.pdf 
https://wyominginstructionalnetwork.com/wp- 


content/uploads/2018/05/MTSS-Implementation- 


Checklist_Literacy.docx 


Source: Authors’ in-depth analysis of tools from the website and document review and consultation with state officials in 2018. 


Names of state tools not included in the in-depth analysis 


Table B4 lists the names of the tools that were not included in the in-depth analysis. The information is organized 
by tool exclusion criteria and presented at the state level. 


Table B4. Names of state tools not included in the in-depth analysis of multi-tiered system of 
supports/response to intervention implementation assessment tools 


State and exclusion 


(org in=}alola) Assessment tool name 


State is using only tools from other states or organizations without modification (n = 6) 


Arkansas e State Capacity Assessment for Scaling Up Evidence-Based Practices (Fixen, Ward, Duda, Horner, & 
Blase, 2015) 
https://implementation.fpg.unc.edu/sites/implementation.fpg.unc.edu/files/NIRN-SCA-v25.3.pdf 

e District Capacity Assessment Version 6.0 (Ward et al., 2015) 
https://miblsi.org/sites/default/files/Documents/Evaluation/Capacity/DCA%206.2%20%20Final%20 
Print%207.30.15%20M1%20Insert.pdf 

e The Reading Tiered Fidelity Inventory Elementary-Level Edition (from Michigan’s Integrated 
Behavior and Learning Support Initiative—MIBLSI) 
https://miblsi.org/sites/default/files/Documents/Evaluation/Fidelity/RTFl/website March2018 R- 
TFI%20Elementary-Level%20Edition v1.3.pdf 

e = The Reading Tiered Fidelity Inventory Secondary-Level Edition (from the MIBLSI) 
https://miblsi.org/sites/default/files/Documents/Evaluation/Fidelity/RTFI/Jan_2018 R- 
TFI_Secondary-Level%20 Edition v1.3 SECURED.pdf 

e The School-wide PBIS Tiered Fidelity Inventory (SWPBIS TFI) (Algozzine et al., 2014) 
https://www.pbis.org/Common/Cms/files/pbisresources/SWPBIS%20Tiered%20Fidelity%20Invento 
ry%20%28TFI%29.pdf 

New Hampshire New Hampshire districts that were part of the SWIFT grant are using SWIFT fidelity tools. 

e —_ Reading Tiered Fidelity Inventory (R-TFl) from MIBLSI 
http://guide.swiftschools.org/Common/Cms/Documents/Inclusive%2520Academic%2520Instructio 
n/Reading%2520Tiered%2520Fidelity%2520Inventory.pdf 

e = SWIFT Fidelity Integrity Assessment (SWIFT-FIA) 
http://www.swiftschools.org/sites/default/files/SWIFT-FIA%202.0%20Fillable.pdf 

New Jersey e —_ Essential Components of RTI Integrity Rubric and Worksheet from the National Center on Response 
to Intervention at American Institutes for Research (AIR) 
https://www.rti4success.org/resource/essential-components-rti-integrity-rubric-and-worksheet 

e¢ ~=Colorado School RTI fidelity rubric 
http://www.rtinetwork.org/images/Colorado School Rtl_Fidelity Rubrics 2.pdf 

e —_ Florida's Self-Assessment of Problem Solving Implementation (SAPSI) 
http://www. florida-rti.org/_docs/AppendixA.pdf 

e = Checklist of Observable Processes during RTI Full Implementation (Developed by Susan L. Hall, Ed.D, 
for the RTI Action Network in 2008) 
http://www.rtinetwork.org/images/content/downloads/get%20started/hall_checklist.pdf 

South Dakota e ~=Michigan’s Integrated Behavior and Learning Support Initiative (MIBLSI) Reading Tiered Fidelity 
Inventory Elementary-Level Edition 
https://miblsi.org/sites/default/files/Documents/Evaluation/Fidelity/RTFl/website March2018 R- 
TFI%20Elementary-Level%20Edition_v1.3.pdf 

e =Michigan’s Integrated Behavior and Learning Support Initiative (MIBLSI) Reading Tiered Fidelity 
Inventory Secondary-Level Edition 
https://miblsi.org/sites/default/files/Documents/Evaluation/Fidelity/RTFIJjan_ 2018 R- 
TFI_Secondary-Level%20 Edition v1.3 SECURED.pdf 

Utah e —_ Essential Components of RTI Integrity Rubric and Worksheet from the National Center on Response 
to Intervention at American Institutes for Research (AIR) 
https://www.rti4success.org/resource/essential-components-rti-integrity-rubric-and-worksheet 
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State and exclusion 
(org in=}alola) Assessment tool name 
Virginia e State Capacity Assessment for Scaling Up Evidence-Based Practices (Fixen, Ward, Duda, Horner, & 


https://implementation.fpg.unc.edu/sites/implementation.fpg.unc.edu/files/NIRN-SCA-v25.3.pdf 


e District Capacity Assessment Version 6.0 (Ward et al., 2015) 
https://miblsi.org/sites/default/files/Documents/Evaluation/Capacity/DCA%206.2%20%20Final%20 
Print%207.30.15%20MI%2OInsert.pdf 

e The School-wide PBIS Tiered Fidelity Inventory (SWPBIS TFI) (Algozzine et al., 2014) 
https://www.pbis.org/Common/Cms/files/pbisresources/SWPBIS%20Tiered%20Fidelity%20Invento 


ry%20%28TFI%29. pdf 
State is using a tool from another state or organization without modification and a general tool (n = 3) 


California e —_ Local Education Agency (LEA) Self-Assessment (Differentiated Assistance Protocols for LEAs — 

Facilitation Guide) 
http://ccsesa.org/wp-content/uploads/2018/05/Facilitators-Guide-v-2.7.pdf 

e Reading Tiered Fidelity Inventory (R-TFI) from MIBLSI 
http://guide.swiftschools.org/Common/Cms/Documents/Inclusive%2520Academic%2520Instructio 
n/Reading%2520Tiered%2520Fidelity%2520Inventory.pdf 

e = SWIFT Fidelity Integrity Assessment (SWIFT-FIA) 
http://www.swiftschools.org/sites/default/files/SWIFT-FIA%202.0%20Fillable.pdf 

e —_ Essential Components of RTI Integrity Rubric and Worksheet from the National Center on Response 
to Intervention at American Institutes for Research (AIR) 
https://www.rti4success.org/resource/essential-components-rti-integrity-rubric-and-worksheet 


New Mexico e Student Assessment Team (SAT) Self-Assessment 
https://webnew.ped.state.nm.us/wp-content/uploads/2018/03/SAT-Self-Assessment.docx 
e —_ Essential Components of RTI Integrity Rubric and Worksheet from the National Center on Response 
to Intervention at American Institutes for Research (AIR) 
https://www.rti4success.org/resource/essential-components-rti-integrity-rubric-and-worksheet 


Ohio e —OIP Implementation Criteria & Rubric 
https://education.ohio.gov/getattachment/Topics/School-Improvement/State-Support-Teams/OIP- 
implementation-Rubric-7-2013.pdf.aspx 

e —_ Reading Tiered Fidelity Inventory (R-TFI) from MIBLSI 
http://guide.swiftschools.org/Common/Cms/Documents/Inclusive%2520Academic%2520Instructio 
n/Reading%2520Tiered%2520Fidelity%2520Inventory.pdf 

State is using a tool from another state or organization without modification and has another tool under revision (n = 1) 


Rhode Island e = Data-based Individualization (DBI) Implementation Rubric and Interview from the National Center 
on Intensive Intervention 
https://intensiveintervention.org/resource/dbi-implementation-rubric-and-interview 


State is using a tool that it developed or adapted from an existing tool and that is publicly available or accessible to the public, 
and state is using a tool from another state or organization without modification (n = 5) 


Illinois e District Capacity Assessment Version 6.0 (Ward et al., 2015) 
https://miblsi.org/sites/default/files/Documents/Evaluation/Capacity/DCA%206.2%20%20Final%20 


Print%207.30.15%20MI%20Insert.pdf 
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State and exclusion 
criterion Assessment tool name 
Michigan e 1. Fidelity assessments: Data to determine how many features of school-wide MTSS are in place in 
schools and the quality of their implementation. 
> School-Wide Positive Behavioral Interventions & Supports (PBIS) Tiered Fidelity Inventory: 
School leadership teams need to know how well PBIS is implemented in the school. The SWPBIS 
TFI combines features from a variety of fidelity assessments. The result is a single, reliable, 
assessment to assist with implementing and sustaining the three tiers of PBIS. 
https://www.pbis.org/Common/Cms/files/pbisresources/SWPBIS%20Tiered%20Fidelity%20Inve 


ntory%20%28TFI%29.pdf 
e  2.Capacity Assessments: Used to determine if the Intermediate School District or district is in a 
position to effectively support schools. 
> Regional Capacity Assessment: Intermediate School Districts assess their capacity to support 
constituent districts within their region to develop an implementation infrastructure to support 
the effective use of an integrated behavior and reading MTSS model. 
https://miblsi.org/evaluation/capacity-assessments/regional-capacity-assessment-%28rca%29 


> District Capacity Assessment: District Implementation Teams need to know how well their 
implementation infrastructure is supporting the effective use and sustainability of the 
components of an integrated behavior and reading MTSS model. 
https://miblsi.org/sites/default/files/Documents/Evaluation/Capacity/DCA%206.2%20%20Final 
%20Print%207.30.15%20M1%20Insert.pdf 

e 3. Scale-Up (Reach) Assessment: Knowing the correct stage of implementation helps Intermediate 

School Districts and districts identify the right training and coaching resources to support 

implementation plans. 

> MIBLSI Stages of Implementation Indicators: Lists the features for each stage of implementation 
for Intermediate School Districts, districts, and schools that are partnering in the MIBLSI model. 
MIBLSI uses indicators that can be pulled directly from the project’s MIBLSI database (MIDATA) 
for efficient assessment. 
https://miblsi.org/evaluation/reach-assessments/miblsi-stages-of-implementation-indicators 


Vermont Some Vermont schools are using: 

e —_ Reading Tiered Fidelity Inventory (R-TFI) from MIBLSI 
http://guide.swiftschools.org/Common/Cms/Documents/Inclusive%2520Academic%2520Instructio 
n/Reading%2520Tiered%2520Fidelity%2520Inventory.pdf 

e = SWIFT Fidelity Integrity Assessment (SWIFT-FIA) 
http://www. swiftschools.org/sites/default/files/SWIFT-FIA%202.0%20Fillable.pdf 

Wisconsin e = The Benchmarks of Quality 
https://www.pbis.org/common/cms/files/pbisresources/BoQ _TeamMemberRating 2010.pdf 

e The School-wide PBIS Tiered Fidelity Inventory (SWPBIS TFI) (Algozzine et al., 2014) 
https://www.pbis.org/Common/Cms/files/pbisresources/SWPBIS%20Tiered%20Fidelity%20Invento 
ry%20%28TFI%29.pdf 

Wyoming e —_ Essential Components of RTI Integrity Rubric and Worksheet from the National Center on Response 
to Intervention at American Institutes for Research (AIR) 
https://www.rti4success.org/resource/essential-components-rti-integrity-rubric-and-worksheet 


State is using only a general tool (n = 1) 
Arizona e The Comprehensive Needs Assessment and Integrated Action Plan for implementation (Appendix B: 
Individual Principle Rubrics for Use by Committees or Teams or Work Groups to Focus on One 
Principle) 
https://cms.azed.gov/home/GetDocumentFile?id=59a45aef3217e11590d74a91 
State-developed tool is not publicly available (n = 1) 
Maryland e = TAP-IT digital portfolio. Requires account access. 
https://marylandlearninglinks.org/digital-portfolio, 
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State and exclusion 
(org n=}alola) Assessment tool name 
Tool is in development (n = 4) 


Indiana A tool is in development. 
lowa A tool is in development. 
Mississippi A tool is in development. 
Tennessee A tool is in development. 
Tool is under revision (n = 3) 

Massachusetts A tool is under revision. 
Hawaii A tool is under revision. 
West Virginia A tool is under revision. 


State verified not having a tool (n = 6) 
Delaware No tool 
District of Columbia No tool 


Georgia No tool 
Missouri No tool 
South Carolina No tool 
Washington No tool 


Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 


Interview selection criteria 


Table B5 shows the four criteria for selecting states to interview, as well as other factors guiding selection for 
interviews and the tool development process. State tools had to meet at least three of the following four criteria 
for the state to be considered for the interview sample: assesses MTSS/RTI practices considered critical by experts, 
includes a focus on reading/literacy, aligns with the Tennessee Department of Education’s (TDOE) purpose for 
districts and schools to assess implementation fidelity or progress in schools for making improvements, and shows 
evidence that the state has used the tool for its intended purpose. In addition, the study team took into 
consideration other factors (tool was pilot or field tested to ensure items and instructions were clear and tool had 
a report of technical adequacy information) and the tool development process (developed a new tool or adapted 
an existing tool). 
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Table B5. Interview selection criteria 
Nevada) 


Criterion Florida Kansas Carolina Oregon Pennsylvania Vermont Wisconsin Wyoming 


Interview selection criteria from proposal (met 3 of the 4 criteria) 


Assesses MTSS/RTI WA J J J J J J JV 
practices considered 

critical by experts 

(at least one of the state's 

tools addressed 75 percent 

of the key RTI components) 


Includes a focus on JV V Vv JV JV V NA Vv 
reading/ 

literacy (includes items 

about reading/ 

literacy, or the tool can be 

viewed through a reading/ 

literacy lens) 


Aligns with TDOE's purpose yf J J J J J J JV 
for districts and schools to 

assess implementation 

fidelity or progress in 

schools for making 

improvements 


Shows evidence that the JV JV JV JV JV JV JV JV 
state has used the tool for 

its intended purpose (for 

example, state reports 

results from tool use or 

offers training on tool use) 

Other factors guiding interview selection (best practices in tool development)? 

Tool was J JV JV JV JV 
pilot or field tested to 

ensure 

items and instructions 

were clear 

Report technical adequacy 

information about tool? 


Tool development process 
Developed a new tool JV V V JV JV V Vv 
Adapted an existing tool JV JV V 


MTSS is multi-tiered system of supports. RTI is response to intervention. TDOE is Tennessee Department of Education. 

Note: Key RTI personnel and leadership at TDOE also provided suggestions about states of interest to them, such as North Carolina and Wisconsin. 
a. Based on Colton and Covert (2007). 

b. Information has been suppressed to protect the anonymity of the two states meeting this selection criteria. 

Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 


Website and document review protocol 


The website and document review protocol describes the information the study team recorded when reviewing 
states’ MTSS/RTI website and documents. 


Sources form 


$01. Source number (state abbreviation, e.g., 01) 


$O2. Source title (title of document) 
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SO3. Source date retrieved 

$04. Source description (if not clear based on the title) 
SO5. Source attachment (if applicable) 

State Name Link to the appropriate state 


State-Coder (The name on the “State” form will appear) 


States form 

S1. State name 

$2. Coder name 

$3. What is the name of the state's RTI or MTSS framework? 


[Select one choice from dropdown menu, or “Add New Choice”] 


S4. Does the state have an assessment tool to measure RTI implementation? 
[Check all that apply in dropdown menu. Depending on how you answer this question, different follow-up 
questions will appear.] 
REMINDER: You will only complete the “Assessments” form in QuickBase if you select “Yes, tool(s) 
tailored to the state.” 


e Yes, tool(s) are tailored to the state 
o IF the above response is selected: 
so §4a. How many assessment tools does the state have for assessing implementation of RTI? 


e Yes, tool(s) are from other states or organizations 

[The tool is NOT state-specific (that is, it is not customized or modified). This includes AIR’s National 

Center on Response to Intervention Fidelity of Implementation Rubric and Worksheet if it has NOT 

been adapted or modified.] 

o This needs to be a tool for assessing RTI or MTSS implementation. It cannot just be a resource. 

o If you only select, “Yes, tool(s) are from other states or organization,” you do NOT need to 
complete the “Assessments” form in QuickBase. You only need to complete the “Sources” and 
“State” forms. 

o IF the above response is selected: 
2  §A4b. Provide the names of the non-state-specific tool(s) and authors. 


o IF “No” is selected: 

= S4c. Please list the websites searched to confirm that an assessment tool to measure RTI 
implementation is not present on the state's website. 

= S4d. Does the state provide information or links about supporting implementation of the RTI 
or MTSS framework? 
oF “Yes”: 

- S$4di. Provide a description of RTI implementation resources, including the names of 
any external resources referenced 


S5. What office/agency/division within the state department of education oversees implementation of RTI? 
[text] 
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S6. Is special education part of the agency/department that helps to implement or oversee implementation 
of RTI? 
[Select yes or no] 


S7. List the name, title, and contact information for lead state RTI or MTSS personnel and a lead state 
representative for special education. 


e Remember to copy and paste the URL for where you found contact information. 
e Please include the name, position, email, and phone number for each contact. 


More detailed instructions: Record the names of three to five individuals who meet the following 
criteria to participate in a potential follow-up interview. The individual should be responsible for 
overseeing RTI implementation and/or tool development, training staff to use the assessment tool, and 
using the RTI implementation assessment tool. Please include a lead state representative for the RTI or 
MTSS program, a lead state representative for special education, a representative in charge of 
overseeing RTI or MTSS implementation, and RTI or MTSS program staff. You may include the name of a 
contractor or consultant if appropriate. 


S8. Does the state provide links to other state RTI or MTSS webpages or resources that are NOT tool- 
specific? If so, list which states. 


S9. Please provide information about the state’s RTI or MTSS framework that you think will be important for 
TN partnership members to know (e.g., information about evaluations of their RTI or MTSS framework; 
grant funding or pilot studies related to RTI or MTSS). Make sure to include URLs. 


$10. Provide any information about whether the state shifted from using an RTI to an MTSS framework, or 
whether the state plans to transition from an RTI to MTSS framework. 


State form questions for the team to discuss 
[Use this space to write down any questions or issues that came up. 
Assessment form 


State Name 

Coder Name 

Al. Assessment tool name 
[text] 


Sources 


AS1. What sources were used for answering questions about this assessment tool? 


[link to ‘Sources’ Form. There will be a dropdown menu of options. Click the appropriate box] 
Basic Information 


ABI1. Is the tool intended to measure implementation at the district, school, or classroom level? 


[Check all that apply in dropdown menu. There is also an option to “Add a New Choice.”] 


ABI2. What is the topical focus of the assessment tool? 


[Check all that apply in dropdown menu. There is also an option to “Add a New Choice.”] 
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ABIS3. 


ABI4. 


ABI5. 


ABI6. 


ABI7. 


What is the publication date of the assessment tool? 


e Use the date on the document (not the file name). Only use the date provided in the file name if you 
CANNOT find the publication date anywhere on the document. 


[text] 


What type of assessment tool is it? 
[Check the appropriate response. There is also an option to “Add a New Choice.” ] 


What is the intended purpose of the assessment tool? 
[text] 


What personnel are intended to administer the assessment tool? 
[text] 


What personnel are intended to use the data collected from the assessment tool? 
[text] 


Domains and rating scale 


ADRS1. 


ADRS2. 


ADRS3. 


ADRS4. 


ADRSS. 


ADRS6. 


ADRS7. 


How many domains (or higher order categories) are included in the assessment tool? 
[Numeric] 


Describe the domains AND components (provide the domain and component labels and indicate the 
number of items in parentheses after each domain). 
[Text] 


How many total items does the assessment tool include? 
[Numeric] 


What type(s) of information or evidence do personnel use to assign ratings/scores on the assessment 
tool (for example, direct observations, interventionists’ data, a team reaches a consensus score)? 
[Text] 


e Onsome assessment tools, there might be an additional column in the tool that provides 
information about the type of evidence raters should be using. 

Describe the assessment tool's rating scale (i.e., copy and paste a description of the labels and 

definitions for rating categories) 

[Text] 


How many points are on the rating scale? 
[Numeric] 


Is information available about how to interpret ratings/scores or about fidelity thresholds for the 
assessment tool? 
[Yes, No] 


Definition of fidelity thresholds - “Fidelity thresholds are numeric scores that are used to define 
different levels of fidelity of a specific indicator. Represented as numeric scales, thresholds quantify the 
extent to which an indicator was enacted with fidelity. This scale can be dichotomous (for example, 0 = 
inadequate fidelity or 1 = adequate fidelity) or it can have a range of fidelity levels (for example, 0 = low 
fidelity, 1 = moderate fidelity, and 2 = high fidelity). Further, the project team can define levels of fidelity 
as low/moderate/high based on the total score across indicators/components or based on the 
percentage of indicators/components fully met.” (Lammert, Heinemeier, Schaff, Fiore, & Howell, 2016, 
p. 114). 
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e IF “Yes” 
© ADRS7i. Copy and paste a text description about how to interpret ratings/score or the fidelity 


thresholds. 
Tool development 
ATD1. Who was involved in the development of the assessment tool for measuring RTI implementation? 
[Text] 
ATD2. Has the assessment tool been piloted? 
[Yes, No] 
e IF “Yes” 
o ATD2aa. If yes, describe pilot testing. (Copy and paste a description, reference source # and page 
#) 
ATD3. What was the process for developing the assessment tool? 
(Copy and paste a text description of the process for developing and/or refining the assessment tool, 
including who was involved and what steps the state took.) 
ATD4. Is there information about the reliability and validity of the assessment tool? IF yes, mark the 
appropriate boxes below. 
[Yes, No] 
e ATD4a. Inter-rater agreement 
o IF checked: 
="  ATD4ai. Describe how inter-rater validity was determined. (Copy and paste a text 
description) 
e ATD4b. Construct validity 
o IF checked: 
=" ATD4bi. Describe how construct validity was determined. (Copy and paste a text 
description) 
e ATD4c. Face validity 
o IF checked: 
=" ATD4ci. Describe how face validity was determined. (Copy and paste a text description) 
Key RTI practices 
AKPI1. Does the tool include items about multiple tiers of instruction? IF yes, check the appropriate boxes 
below. 
[Yes, No] 
AKP2. Does the tool include items about data-based decision-making? IF Yes, check the appropriate boxes 
below. 
[Yes, No] 
AKP3. Does the tool include items about staffing roles/infrastructure for implementation/leadership? IF Yes, 
please check the appropriate boxes below. 
[Yes, No] 
AKP4. How does the state define “evidence-based interventions” (that is, what criteria are used)? 


[Text] 
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Supporting assessment tool use 
ATU1a. What personnel are trained to administer the assessment tools? 


e There needs to be explicit evidence stating that people are TRAINED to ADMINISTER the tool. 
[Text] 


ATU1b. What training or support do personnel receive on the administration of the assessment tool? 
[Check all that apply in dropdown menu. There is also an option to “Add a New Choice.”] 


e Training needs to be specific to the tool (do NOT include training that is about any aspect of the 
state’s RTI or MTSS framework). 
o IF you select “In-person training”: 
"  ATU1bi. Describe how often the training/support is offered and for what length of time 
(e.g., days/hours): in person training 
© IF you select “1-on-1 coaching”: 
"  ATU1bii. Describe how often the training/support is offered and for what length of time 
(e.g., days/hours): 1-on-1 coaching 
o IF you select “technical assistance”: 
"  ATU1biii. Describe how often the training/support is offered and for what length of time 
(e.g., days/hours): technical assistance 
o IF you select “Webinars”: 
"  ATUtbiv. Describe how often the training/support is offered and for what length of time 
(e.g., days/hours): webinars 
o IF you select “Online self-paced course”: 
="  ATU1bv. Describe how often the training/support is offered and for what length of time 
(e.g., days/hours): online self-paced courses/modules 
o IF you select “Other”: 
=" ATU1b_oth. Describe “Other” 
="  ATU1bvi. Describe how often the training/support is offered and for what length of time 
(e.g., days/hours): Other 


ATU2a. What personnel are trained to use the data collected from the assessment tool? 


e There needs to be explicit evidence stating that people are TRAINED to USE the data collected from 
the tool. 
Definitions: Data use includes managing or uploading data to a system, preparing data to facilitate 
conversations, and analyzing or interpreting data for its intended purpose. 

[Text] 


ATU2b. What training or support do personnel receive regarding using the RTI Implementation assessment tool 
data? 


[Check all that apply in dropdown menu. There is also an option to “Add a New Choice.” ] 
e IF you select “In-person training”: 
© ATU2bi. Describe how often the training/support is offered and for what length of time (e.g., 
days/hours): in person training 
e IF you select “1-on-1 coaching”: 
© ATU2bii. Describe how often the training/support is offered and for what length of time (e.g., 
days/hours): 1-on-1 coaching 
e IF you select “technical assistance”: 
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ATU3. 


ATU4. 


ATUS. 


ATU6. 


ATU7. 


© ATU2biii. Describe how often the training/support is offered and for what length of time (e.g., 
days/hours): technical assistance 
e IF you select “Webinars”: 
© ATU2biv. Describe how often the training/support is offered and for what length of time (e.g., 
days/hours): webinars 
e IF you select “Online self-paced course”: 
© ATU2bv. Describe how often the training/support is offered and for what length of time (e.g., 
days/hours): online self-paced courses/module 
e IF you select “Other”: 
o ATU2b_oth. Describe “Other” 
© ATU2bvi. Describe how often the training/support is offered and for what length of time (e.g., 
days/hours): Other 


What types of resources and documents are available to support personnel in using the state's RTI 
implementation assessment tool (e.g., guidance document, technical manual)? List the names of the 
documents here, AND document in the Sources Form. *Copy and paste the URL* 


What are the state's guidelines for how often or when the assessment tool should be used? 


[Check response from dropdown menu.] 


e IF you select “Other”: 
o ATU4i. Describe “Other” 
e IF you select “Additional details”: 
o ATU4i. Provide additional details 
Who receives or has access to the results of the assessment tool for measuring RTI implementation? 


(Copy and paste a description OR state this information is not available) 
How are states supposed to use the information collected through the RTI implementation assessment 
tool? 


(Copy and paste a description OR state this information is not available) 


e = This can include what they’ve already done with the assessment tool results, OR it may be a 
description of what should happen with the results from the implementation assessment tool. 
How is technology used to support RTI implementation data collection and use? 


[Check all that apply in dropdown menu. You can also “Add a New Choice”] 


Interview criteria 


This section is intended to identify whether the state’s assessment tool meets the criteria for state personnel to 
participate in an interview. Check each criterion that is met. If you are unsure, please note this in the area for 


“Assessment form questions for the team to resolve.” 


e 1C1. The assessment tool measures RTI practices considered critical by RT| experts (Gersten et al., 
2008) and the research literature. 

e C2. The assessment tool includes a focus on reading and/or literacy, because improving literacy in 
the early elementary grades is one of the Tennessee Early Literacy Partnership’s goals. 

e 1C3. The assessment tool aligns with TDOE’s purpose for districts and schools to assess fidelity of 
implementation in schools for making improvements. 

e  1C4. There is evidence that the state has used the assessment tool for its intended purpose (for 
example, the state reports results from assessment tool use, the state has offered training on how 
to use the assessment tool). 
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Questions for the team 

Assessment form questions for the team to resolve 

[Use this space to write down any questions or issues that came up, or recommendations for how we may want 
to modify the form] 


Methods for coding and analyzing key multi-tiered system of supports/response to intervention 
practices 


This section describes the process that the study team used to define key MTSS/RTI practices; provides definitions 
of components, subcomponents, and dimensions; describes the systematic process for coding key MTSS/RTI 
practices; and describes the analyses used to examine whether key MTSS/RTI practices varied by state framework 
terminology. 


Process for defining key practices. To define MTSS/RTI practices for measuring implementation, the study team 
referred to established practice guidelines, supporting research literature, state instruments and instruments 
from national organizations, and technical expert review. To identify the key practices to include in the review of 
tools, the study team followed a similar rationale to that used for Evaluation of the Response to Intervention 
Practices for Elementary School Reading (Balu et. al., 2015; see appendix A for supporting research literature). This 
rationale identifies the types of practices for inclusion in MTSS/RTI frameworks: 


e Practice guidelines that experts recommend, such as administering universal screening at least twice a year 
to all students, using reliable and valid screening tools, varying instructional intensity by varying group size 
and dosage, monitoring tier 2 students monthly, and monitoring tier 3 students weekly. 


e Practices critical to all MTSS/RTI models, such as using data to make decisions about targeting interventions, 
to monitor progress, to tie interventions to core curriculum, and to individualize tiers 2 and 3. 


e Practices that support MTSS/RTI implementation, such as staff supports. 


The study team then referred to the research literature (see appendix A) to define the key MTSS/RTI practices 
more fully. 


Also included was “using evidence-based practices” as referenced in federal legislation. Both the Every Student 
Succeeds Act (2015) and the Individuals with Disabilities Education Improvement Act (2004) require states to 
support districts and schools in using instructional programs and interventions that have evidence to show that 
they are effective at producing results and improving student outcomes when implemented. 


Next, the study team reviewed state instruments (for example, those in Florida, New Hampshire, and Wisconsin) 
and instruments from national organizations (for example, National Center for Response to Intervention and RTI 
Action Network) to identify the practices and how they were organized. From that review, the study team 
identified the four broad components (administer assessments, offer multiple tiers of instruction and 
intervention, support data-based decisionmaking, and support infrastructure practices for MTSS/RTI 
implementation) and developed a hierarchy of the subcomponents and dimensions under the four key 
components. 


Finally, as part of the technical review, experts in MTSS/RTI, positive behavioral interventions and supports, 
reading research, and data-based decisionmaking reviewed the MTSS/RTI key practices. They concluded that the 
rationale for the key practices was well documented. 


Definitions of components, subcomponents, and dimensions. The framework for coding key MTSS/RTI practices is 
hierarchical. At the highest level, it includes four broad components or main features of MTSS/RTI (for example, 
administer assessments). These components comprise more detailed subcomponents or specific aspects within a 
component (for example, administer universal screening measures). For coding purposes, a component was 
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considered to be addressed only if it comprised all associated subcomponents. For example, a tool was considered 
to address the component “administer assessments” only if it included items that addressed both of its 
subcomponents: administer universal screening measures and administer progress monitoring measures. 


Subcomponents are then broken down further into dimensions that help define and measure the subcomponents 
(for example, use reliable and valid screening tools). 


An item is a statement of performance on the measurement tool. 


Systematic process for coding key practices. The study team used a systematic and replicable process for reviewing 
and coding each MTSS/RTI implementation assessment tool. Decision rules were used to code for the presence or 
absence of each key MTSS/RTI component, subcomponent, and dimension (table B6). 


Tools may have directly or indirectly included a specific practice. The study team coded for the presence of 
components, subcomponents, and dimensions if the tool either directly stated that criteria should be in place by 
using item terminology that was identical or similar to the coding scheme established for the study or indirectly 
indicated that criteria should be in place by embedding the criteria as part of a different item that covered other 
MTSS/RTI practices. 


Below are two examples that illustrate how the study team applied these decision rules to the MTSS/RTI 
subcomponent key practice “administer universal screening measures.” Example 1 shows tool documentation 
that directly states that universal screening should be in place, whereas example 2 illustrates the use of language 
that indirectly indicates this requirement. 


Example 1: North Dakota’s tool has a specific item called “universal screening,” and the criteria for the ideal 
level of implementation (that is, a rating of 5 compared with a rating of 1 or 3) states: All of the following are 
met: (1) screening is conducted for all students (i.e., is universal); (2) procedures are in place to ensure 
implementation accuracy (i.e., all students are tested, scores are accurate, cut points/decisions are accurate); 
and (3) a process to screen all students occurs more than once per year (e.g., fall, winter, spring). (p. 1) 


In this case the tool directly describes the MTSS/RTI subcomponent practice “administer universal screening 
measures” and would be coded as present. The tool also directly describes the dimensions of “administer at 
least twice a year” and “include all students,” which would also be coded as present. 


Example 2: Kentucky’s tool does not have an item labeled “administer universal screening measures” but does 
have the following item: Diagnostic Assessments and Progress Monitoring—after students are identified with 
a universal screener, a diagnostic assessment is in place to accurately identify areas of strength and weakness 
so interventions can be targeted and consistent progress monitoring matches the targeted skill and is used to 
drive instruction. (p. 2) 


Although Kentucky’s tool does not directly state that universal screening must be in place, the item indirectly 
conveys this because a diagnostic assessment would be conducted only after the student is identified from a 
universal screening. Therefore, the study team coded as present the subcomponent “administer universal 
screening measures” for this tool. 


A third example shows how the study team applied coding procedures when a tool could meet criteria for a 
component, subcomponent, or dimension if any part of an item directly or indirectly specifies a key MTSS/RTI 
practice. 


Example 3: Michigan’s R-TFI [Elementary] tool (p. 40) has an item that states, Staff collect progress monitoring 
data with fidelity. The description of this item at the ideal implementation level (that is, the criteria for 
receiving 2 points for this item on a scale of 0—2) states, The frequency of progress monitoring is at least once 
per week for students receiving Tier 3 interventions, and every other week for students receiving Tier 2 
interventions. 
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The study team coded this item as meeting the key criteria for the dimension, “administer progress monitoring 
measures monthly at tier 2 and weekly at tier 3,” even though Michigan’s overarching item is broader than 
this study’s key practice. That is, Michigan’s item is about staff collecting progress monitoring data with 
fidelity, whereas this study’s practice focus is on collecting progress monitoring data and on how frequently 


to collect it (figure B1). 


Figure B1. Screenshot from Michigan’s Reading Tiered Fidelity Inventory [Elementary] tool 


Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 
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Table B6. Decision rules for applying multi-tiered systems of supports/response to intervention practice codes to states’ assessment tools 


Assessment tool 


coding category 


Coding decisions: Tool was coded “yes” if... 


Example? 


Administer 
assessments 


Administer universal 
screening measures 


Establish end-of-year 
benchmarks 
Administer at least 
twice a year 


Use reliable and valid 
screening tools 


Include all students 


Component 


Subcomponent 


Dimension 


Dimension 


Dimension 


Dimension 


It addresses both universal screening practices 
and progress monitoring practices 
subcomponents. 

Items directly or indirectly state that universal 
screening assessments should be administered. 


Items directly or indirectly state that 
benchmarks or cutscores should be established. 
Items directly or indirectly state that screening 
should occur at least twice a year, such as ina 
description of the schedule, a description of the 
schedule referencing two (or more) 
administrations, or the use of terminology 
conveying more than one (such as multiple). 


Items directly or indirectly state that universal 
screening measures should be reliable and valid 
through use of the following terminology: 

e Reliable or valid. 

e —- High-quality. 

e ~—-_ Evidence-based. 

e —_- Research-based. 

Items directly or indirectly state that screening 
should occur for all students. 


Universal screening system assesses strengths and challenges of 
all students in academic achievement, talents, and behavior 
(Arkansas, p. 54). 

Delivery of universal curriculum and instruction is determined by 
grade-level/course benchmarks (Wisconsin [All Staff], p. 2). 
Item: We closely, frequently, and systematically assess students 
and make regular and effective use of the data we collect to 
make decisions. 

Description of implementation: The assessment system provides 
information about all students at least 3 times each year to 
determine progress toward standards-based objectives and to 
identify students who are at risk and/or not responding to 
instruction/intervention (Vermont [Self-Assessment], p. 23). 
Evidence indicates that the screening tools are reliable, 
correlations between the instruments and valued outcomes are 
strong, predictions of risk status are accurate, and staff is able to 
articulate the supporting evidence (North Dakota, p. 1). 

Note: Another item specifies administration of universal 
screening. 


My school/district has established a regular schedule for 
screening ALL students in grades K-4 a minimum of three times 
per year (New York [Elementary], p. 7). 
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Assessment tool 
coding category 
Administer progress Subcomponent 


monitoring measures 


Administer progress Dimension 
monitoring measures 
monthly at tier 2 and 


weekly at tier 3 


Use reliable and valid Dimension 
progress monitoring 


tools 


Coding decisions: Tool was coded “yes” if... 
Items directly or indirectly state that progress 
monitoring assessments should be 
administered. 


Items directly or indirectly state that progress 
monitoring assessments should be 
administered at least monthly at tier 2 and 
weekly at tier 3. General descriptions of how 
often progress monitoring measures should be 
administered convey that progress monitoring 
never occurs less often at any tier. For example, 
items using terminology that does not quantify 
the number of times to measure at each tier 
(such as “measures should be administered 
based on research”) were not coded as meeting 
criteria. 

Items directly or indirectly state that progress 
monitoring measures should be reliable and 
valid through use of the following terminology: 
e Reliable and valid. 

e — High-quality. 

e ~—- Evidence-based. 

e —_ Research-based. 


Example? 

Teachers use progress monitoring data to evaluate instructional 
effectiveness and to be informed about the changes necessary to 
better meet students’ needs (e.g., Data Teams) (Connecticut, p. 
4). 

Both of the following conditions are met: (1) progress monitoring 
occurs at least monthly for students receiving Tier 2/Strategic 
intervention and at least weekly for students receiving Tier 
3/Intensive intervention; and (2) procedures are in place to ensure 
implementation accuracy (i.e., appropriate students are tested, 
scores are accurate, decision-making rules are applied 
consistently) (North Dakota, p. 3). 


The BLT [building leadership team] administers high quality, 
student-level progress monitoring tools to analyze student 
performance in academic and behavioral domains (Colorado 
[BLT], p. 2). 
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Assessment tool 
coding category 


Offer multiple tiers of Component 
instruction and 

intervention 

Offer tier 1 instruction Subcomponent 
Use evidence-based Dimension 
programs 

Tie to standards-based Dimension 
curriculum 

Address differentiation Dimension 
of instruction 

Offer tier 1 to all Dimension 


students 


Coding decisions: Tool was coded “yes” if... 
It addresses at least three tiers of instruction 
and intervention. 


Items directly or indirectly state that tier 1, also 
referred to as core instruction, should be in 
place. Tier 1 was coded as being offered if the 
tool explicitly refers to tier 1 or describes 
elements of tier 1 instruction, including high- 
quality classroom-based instruction, use of 
evidence-based programs, programs tied to 
standards-based curriculum, and use of 
differentiated instruction as part of a core 
program. 

Items directly or indirectly state that an 
evidence-based program should be used in the 
core curriculum/tier 1 through use of the 
following terminology: 

e ~—-_ Evidence-based. 

e —_Research-based. 

e —_ Research-validated. 

Items directly or indirectly state that the core 
curriculum should be tied to state or national 
content standards. 

Items directly or indirectly state that 
differentiation of instruction should occur 
during core instruction/tier 1. 


Items directly or indirectly state that tier 1 
should be offered to all students. 


Example? 


Tiered reading instruction includes: 

e Tier 1 evidence-based core curriculum 

e = Tier 1 differentiated instruction 

e Tier 2 individualized and small group interventions 
e = Tier 3 individualized and small group interventions 
(Oklahoma, p. 6). 


Use of a research-validated core reading program, core math 
program, writing program, and behavior [program] at each 
elementary or secondary school identified as RTI ready with 80% 
success rate (Arkansas, p. 53). 


The core curriculum (reading and math) is aligned with the North 
Dakota state standards (North Dakota, p. 7). 


Core Instruction with Differentiation—a variety of teaching 
strategies to address diverse learners (e.g., questioning 
techniques, role playing, manipulatives, literature circles, 
movement activities, project-based learning, flexible groups, etc.) 
(Kentucky, p. 2). 

The BLT manages and monitors the provision of high-quality, 
prevention-based, Universal (Tier 1) supports that are accessible 
to every student and meet the needs of most students (Colorado, 
p. 4). 
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Offer tier 2 
intervention 


Require consideration 
of group size and 
dosage 


Tie to core curriculum 


Use evidence-based 
interventions 


Provide individualized 
instruction 


Assessment tool 


coding category 


Subcomponent 


Dimension 


Dimension 


Dimension 


Dimension 


Coding decisions: Tool was coded “yes” if... 


Items directly or indirectly state that students 
who are not meeting grade-level expectations 
should be provided increased instructional 
intensity. Tier 2 was coded as being offered if 
the tool explicitly refers to tier 2 or describes 
varying the instructional intensity, such as 
providing an intervention, individualizing 
instruction, group size, and/or dosage. If a tool 
conveys that there are at least two tiers but 
does not distinguish a third tier, the tool 
provides credit for offering tier 2 but not tier 3. 
Items directly or indirectly state that group size 
and dosage should be adjusted to increase 
intensity in tier 2. 


Items directly or indirectly state that tier 2 
instruction is tied or linked to core curriculum, 
sometimes referred to as tier 1. 


Items directly or indirectly state that 
interventions at tier 2 should be evidence 
based, using terminology such as: 

e ~—- Evidence-based. 

e —_Research-based. 

e Research-validated. 

Items directly or indirectly state that students 
should be provided instruction that meets their 
specific needs. 


Example? 

Tier 2 (Supplemental) academic practices exist that include 
strategies addressing integrated common student needs, are 
linked to Tier 1 instruction, and are monitored using 
assessments/data sources tied directly to the academic, behavior, 
and social emotional skills taught (North Carolina, p. 10). 


Additional or supplemental instructional time (at least 20-30 


minutes per session, 3-4 times per week) is offered in addition to 
the 90+ minutes provided in core reading instruction. 

Tier 2 interventions are delivered in small group formats (no more 
than 5 or 6 students per group); groups are homogenous (New 
York, p. 5). 


Tier 2 (Supplemental) academic practices exist that include 


strategies addressing integrated common student needs, are 
linked to Tier 1 instruction, and are monitored using 
assessments/data sources tied directly to the academic, behavior, 
and social-emotional skills taught. (North Carolina, p. 10) 


For assessment of learning at the SELECTED AND INTENSIVE 


levels, use evidence-based interventions for students not meeting 
benchmarks (Wisconsin, p. 1). 


Diagnostic assessment data are used to inform decisions about 


strategic and intensive reading interventions for individual 


students (Kansas [Instructional Staff Survey], p. 1). 
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Assessment tool 


coding category 


Coding decisions: Tool was coded “yes” if... Example? 


vc-4d 


Offer tier 3 Subcomponent Items directly or indirectly state that students Students who require even more intensive or specialized 

intervention who are considered far below grade level instruction receive additional intervention services and supports, 
should be provided intensive, individualized including, but not limited to, Special Education services [Tier 3] 
instruction. Tier 3 was coded as being offered if | (Vermont, p. 19). 
the tool explicitly refers to tier 3 or describes 
individualizing instruction or increasing intensity 
beyond tier 2 practices. 

Require consideration Dimension Items directly or indirectly state that group size =‘ Intervention intensity is determined through review of data for 

of group size and and dosage should be adjusted to increase considerations about frequency, duration, size of grouping, and 

dosage intensity in tier 3. effectiveness of instruction/intervention (Connecticut, p. 1). 

Tie to core curriculum Dimension Items directly or indirectly state that tier 3 Element: Tier 3 (intensive) academic practices exist that include 
instruction should be tied or linked to core integrated strategies that are developed based on students’ 
curriculum, sometimes referred to as tier 1. needs, are aligned with Tier 1 and Tier 2 instructional goals and 

strategies, and are monitored using assessments/data sources 

that link directly to skills taught. 

Description: 

Tier 3 strategies incorporate all of the following: 

e developed based on students’ needs across academic, 
behavior and social-emotional domains 

e = aligned with Tier 1 and Tier 2 instruction 

e linked to behavior and social- emotional content/instruction 

e monitored using assessments/data sources that link directly 
to the skills taught 

(Illinois, p. 16). 

Use evidence-based Dimension Items directly or indirectly state that Student needs are met via design and delivery of increasingly 

interventions interventions at tier 3 should be evidence robust differentiated instruction that is aligned across the tiers to 
based, using terminology such as: grade level standards. Empirically-supported instructional 
e ~~ Evidence-based. strategies and standard treatment/protocol interventions are 
e Research-based. used to facilitate student response to instruction and expedite 
e _Research-validated. learning/growth. (Pennsylvania, p. 10) 

Note: Other items in the tool establish practices of multiple tiers. 

Provide individualized Dimension Items directly or indirectly state that students Student Intervention Plans—plans are individualized and 


instruction 


should be provided instruction that meets their 


specific needs. 


constantly changing as the needs of the student change 
(Kentucky, p. 2). 
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Assessment tool 


coding category 


Support data-based Component 
decisionmaking 

Establish data rules Subcomponent 
Use tier 2 and 3 Dimension 


progress monitoring 

data to determine 

responsiveness to 

interventions 

Use multiple sources of | Dimension 
data to inform 

decisions 


Use analysis of tier 2 Dimension 
and 3 progress 

monitoring data 

inclusive of slope of 

improvement or 

progress toward 


attainment of a goal 


Coding decisions: Tool was coded “yes” if... 
It addresses data-based decisionmaking. 


Items directly or indirectly state that there 
should be explicit guidelines for using data to 
make decisions about student performance. 


Items directly or indirectly state that tier 2 and 
3 data (sometimes referred to as progress 
monitoring data) should be used to determine 
responsiveness to interventions. 


Items directly or indirectly state that multiple 
sources of data should be used to make 
decisions regarding students. Data may be 
within a content area (for example, multiple 
literacy assessments) or across content areas 
(for example, examining literacy and behavior 
assessments). 

Items directly or indirectly state that tier 2 and 
tier 3 data (sometimes referred to as progress 
monitoring data) should be reviewed in the 
context of progress toward a specific goal, 
sometimes presented as a rate of improvement. 


Example? 


Item: Processes, procedures, and decision-rules are established 
for data-based problem-solving. 

Description of implementation: Processes, procedures, and 
decision-rules needed to engage in data-based problem-solving 
are developed and existing structures and resources are 
incorporated, and The steps of problem-solving; procedures for 
accessing, submitting, and using data; and decision-rules needed 
to make reliable decisions are communicated to staff, and Data- 
based problem-solving processes, procedures, and decision-rules 
are refined based on data and feedback from staff, schedule 
changes, and resource availability (Florida, p. 4). 

For assessment of learning at the SELECTED AND INTENSIVE 
levels, we frequently review progress-monitoring data to gauge 
whether students are making adequate progress in response to 
the interventions/challenges and adjust accordingly 

(Wisconsin, p. 2). 

The school uses a variety of data sources to design intensive math 
intervention plans (Minnesota, p. 45). 


Both conditions are met: 1) decisions about RTI are based on 
reliable and valid PM [progress monitoring] data to reflect rate of 
improvement or final status; 2) decisions are implemented 
accurately (Idaho, RTI Rubric Excel sheet, row 58, columns R-V). 
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Assessment tool 


Support infrastructure 
practices for MTSS/RTI 
implementation 

Establish building-level 
implementation teams 


Allocate time for teams 
to meet 


Evaluate individual 
student progress using 
screening and progress 
monitoring data 


Use a problem-solving 
approach for 
decisionmaking 

Offer coaching to 
support 
implementation (for 
example, to 
understand data) 


coding category 
Component 


Subcomponent 


Dimension 


Dimension 


Dimension 


Subcomponent 


Coding decisions: Tool was coded “yes” if... 

It addresses both staffing for data-based 
decisionmaking practices and availability of 
coaching to support implementation practices. 
Items directly or indirectly state that staff 
should be organized to monitor MTSS/RTI 
implementation and make data-based decisions 
(that is, have established building or grade-level 
teams for evaluating data). 

Items directly or indirectly state that time 
should be set aside for building- or grade-level 
teams to meet to discuss MTSS/RTI 
implementation. 

Items directly or indirectly state that building- 
or grade-level teams should meet to evaluate 
screening and progress monitoring data 
(sometimes referred to as assessment data) of 
individual students. 


Items directly or indirectly state that a problem- 
solving approach for decisionmaking should be 
used by building- or grade-level teams. 

Items directly or indirectly state that coaching 
should be available to support implementation 
of MTSS/RTI. Coaching support may be general 
or address specific topics related to MTSS/RTI. 


MTSS is multi-tiered systems of supports. RTI is response to intervention. 


a. The examples come from the tools listed in table B4. 


Example? 


School Implementation Team—team members understand roles 
and meet regularly to discuss school wide implementation (e.g., 
schedule, data, criteria for tier movement, resources, etc.) 
(Kentucky, p. 2). 


Schedules provide adequate time for staff to engage in 
collaborative, data-based problem-solving and decision-making 
(Illinois, p. 10). 


Staff understand and have access to academic, behavior, and 
social-emotional data sources that address the following 
purposes of assessment: 
e identify students at-risk academically, socially, and/or 
emotionally 
e = determine why student is at-risk 
e = =monitor student academic and social-emotional 
growth/progress 
e — inform academic and social emotional 
instructional/intervention planning 
e = determine student attainment of academic, behavior 
and social emotional outcomes 
(Florida, p. 11). 
My building leadership team regularly engages in formal problem 
solving using data to make decisions (Kansas [Administrator 
Survey], p. 3). 
Coaching is used to support MTSS implementation (North 
Carolina, p. 5). 


Source: Author-developed coding scheme and decision rules for analyzing states’ MTSS/RTI implementation assessment tools for evidence-based practices in 2018. 


Analyses to examine whether key MTSS/RTI practices varied by state framework terminology 


The study team also examined whether the percentage of key MTSS/RTI practices included in each tool varied 
according to whether the state referred to its framework as MTSS or RTI. No consistent differences emerged. For 
example, 6 of the 9 states that reported using an RTI framework used a tool that addressed both academics and 
behavior (see appendix E). Because some state personnel use the terms MTSS and RTI interchangeably, this lack 
of difference in tool content is not surprising. 
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Appendix C. Interview protocol 

This appendix contains the interview protocol that the study team used to conduct semi-structured interviews 
with state education officials in eight states to learn more about the how states adapted or developed their 
assessment tools for multi-tiered system of supports (MTSS)/response to intervention (RTI) and supported schools 
and districts in using the tools and resulting data. 


Introduction and purpose of interview 

Thank you for your willingness to participate in this interview. We are talking with you about how your state 
selected, adapted, or developed an assessment tool for measuring response to intervention (RTI) implementation 
[or MTSS]° and is using the results from the assessment tool. We’d like to learn about what is working well in 
addition to the challenges and how you’re working to overcome them. We believe information about your 
experiences will be valuable for other states interested in developing and using an assessment tool for measuring 
RTI implementation. This interview will take approximately 75-90 minutes. 


Consent process 


There are no anticipated or known risks in participating in this study. Your participation in the interview is 
voluntary. If you decide to participate, you may discontinue your participation in the interview at any time without 
penalty of any kind. 


Do you agree to participate in this phone interview? 


We will also be taking notes and recording the interview for our own note-taking purposes. Again, these notes 
and recordings will not be shared with anyone outside the study team. Do you consent to be recorded? 


€ Denotes that the interviewer will ask for additional documents. 


Description and development of the assessment tool 


In our first set of questions, we are interested in understanding the process you went through to [select, modify, 
or develop] the [assessment tool], such as the factors you needed to consider (for example, purpose, users, 
process). 


1. For what purposes was the assessment tool selected or developed in your state? 


(If needed, probe for whether the intended purpose of the assessment tool is for self-assessment, 
improvement, accountability, or implementation fidelity. Also probe if the assessment tool is intended to 
support special education designation based on data-based decisionmaking, or evidence-based programs 
for the general education population.) 


2. What factors served as an impetus for having an assessment tool? 


(If needed, probe for factors including state guidelines, an advisory board, program evaluation, concerns 
about districts and schools not implementing RTI with fidelity, or districts and schools seeking guidance 
about RTI implementation.) 


6 Interviewers will use the term “RTI” or “MTSS” depending on the terminology used by the state. 


REL 2020-017 Cc-1 


3. Was the assessment tool selected from an existing assessment tool? 


O 
O 


Yes {Go to Q3a, and skip Q3b} 
No {Go to Q3b} 
If yes, was the assessment tool adapted? 
i. How was the assessment tool modified? 
ii. Why was it necessary to modify the assessment tool? 
iii. How many years has the state been using this assessment tool? 
If no, what resources did the state use to develop the assessment tool? 
i. Why did the state choose to develop a new measure? 


ii. Did the state receive support from outside experts? If yes, how did they support the 
development work? 


iii. How many years has the state been using this assessment tool? 


4. What factors did the state consider when developing, adapting, or selecting an assessment tool? 


(First let the respondent answer without any prompts. Then ask 4a—4c if these topics were not mentioned.) 


a. 


Was burden on district or school staff a consideration in developing/selecting the assessment 
tool (for example, number of items, time commitment)? 


Was alignment with the state’s MTSS/RTI recommended practices a consideration in 
developing/selecting the assessment tool? 


Was technology a consideration in selecting the assessment tool? 


i. If yes, how? 


€ Is there a document that summarizes the selection (or development) process? 


If yes, could you share a copy? 


5. How did the state decide on what key MTSS/RTI practices are represented in the assessment tool? 
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How does your state define “evidence-based” interventions? 
How is this definition applied to the selection of programs and practices? 


What type of evidence is required to provide ratings or scores on the assessment tool (for 
example, direct observation; collection of interventionist’s data; coaches’ records; teacher self- 
report)? 


C-2 


6. How did the state decide on the type of rating scale? 
a. What types of fidelity thresholds have been established, if any? 
[If fidelity thresholds have NOT been established, skip to Q6c.] 
b. How were fidelity thresholds established? 
[Skip to Q7.] 
c. [lf thresholds have not been established] Why haven't fidelity thresholds been established? 
(Probe for factors like insufficient evidence, lack of technical skill.) 
7. Was the assessment tool pilot- or field-tested? 
O Yes {Go to Q7a and skip Q7e} 
O No {Go to Q7e} 
a. Can you briefly describe the process? 
b. What did you learn from the results? 


c. Were the results used to modify the content or procedures of the assessment tool? If yes, how 
so? 


d. How was the pilot or field test supported (for example, hired a contractor, schools and/or 
teachers received stipends, release time)? 


e. Why is information about pilot testing not available (is it because of cost, time, expertise)? 


€ Is there a copy of the pilot and/or field test findings you could share with us? 


8. [IF information about the assessment tool’s technical properties exists based on the 
website/document review summary sheet and verification process. If this information is not available, 
skip to Q9.] We located some information about reliability and validity and have a few questions to help 
us understand the process. 


a. What information does the state have on the assessment tool’s inter-rater reliability (or inter- 
rater agreement) across administrators? 


b. What information does the state have on the construct validity of the domains? 


c. What information does the state have on the face validity of the items on the measure? 


© Is there a copy of how (any of) these technical properties of the measure were reported that 
you could share with us? 


& Are there updated documents with this information available you could share with us? 
[Then skip to Q10.] 


9. [IF information about the assessment tool’s technical properties does not exist based on the 
website/document review summary sheet and verification process.] Why is information about the 
assessment tool’s reliability and validity not available (is it because of cost, time, expertise)? 


10. What were the biggest challenges with selecting or developing an assessment tool to measure RTI 
implementation? 
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Description of how states use the assessment tool 


We’d also like to learn about how your state supports districts and or schools in using the assessment tool to 
measure MTSS/RTI implementation. 


11. Are all the [districts or schools] in the state using the assessment tool? 
a. If not, how do [districts or schools] opt out of administering the assessment tool? 
b. How much autonomy do districts have in modifying or adapting the assessment tool? 


12. How did the state roll out the assessment tool for use by [name the intended user based on the 
document review]? 


a. What was the outcome from this approach? 

b. How did the state follow up with [districts, schools] after the initial rollout? 
c. What would the state retain about this approach? 

d. What would the state change about this approach? 

e. What challenges did the state face with this approach? 


13. Why did the state decide to train [type of personnel identified from website/document review and 
verification process]? Were other types of personnel considered for training, and if so, why were they 
not included? 


[Note — You are not asking for the names of specific individuals, but rather types of personnel (for example, 
district administrator, schoo! administrator, regional! trainer).] 


14. After the initial rollout, how did the state provide support for continued use of the assessment tool? 
a. How effective does the state think the training is for supporting end users? 
b. If continued support is not provided, why not (for example, cost, time)? 

15. How has your state supported districts and schools to use the assessment tool? 


a. Has the state used existing in-state professional development networks? Trainer of trainer 
models? Pilot sites that serve as models to showcase? 


16. Is the training/support offered the same for all schools and districts statewide or is it school or district 
specific? 


a. If different, how do they differ and who tailors them? 
b. Can you provide an example of tailored support? 


We’d like to understand how the state uses this information from the administration of the MTSS/RTI 
implementation assessment tool. 


17. Who receives the results from the administration of the MTSS/RTI implementation assessment tool? 


18. Does the state specify how the findings from the administration of the assessment tool should be 
shared? And with whom? 


19. How are the results reported or displayed (for example, report on a data dashboard)? 


a. Is there an example to share? 
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20. What are the next steps after findings/results are shared (for example, create an action plan for the 
school or district, target professional development for many or for a few, shape policy or changes to the 
MTSS/RTI approach)? 


a. Is there an example to share? 


21. What are the benefits and challenges of using the data collected from the assessment tool (for example, 
challenges in interpreting the data to know what to do next)? 


22. Has your state used the assessment tool to evaluate implementation of the state’s RTI or MTSS 
framework? Or are you currently evaluating implementation of the RTI or MTSS framework? 


a. If yes, in what year? 


b. Are there specific districts that have used the assessment tool for evaluation purposes? If yes, 
which specific districts? 


© If yes, is the report available that you would be able to share? 


Lessons learned and recommendations 


This is our final set of questions! We’d like you to reflect on how your state has supported districts and schools 
to administer the assessment tool for measuring MTSS/RTI implementation and to use the data collected from 
the assessment tool. 


23. Are there lessons learned or recommendations you can share with other states about the selection, 
implementation, and use of the assessment tool for measuring implementation of MTSS/RTI? 


24. Are there any districts in your state that may serve as exemplars for how to effectively collect and use 
the data from the assessment tool to improve MTSS/RTI implementation? 


25. Has your state transitioned from using an RTI to an MTSS framework? 
a. If yes, did this have implications for changing the tool? If yes, how so? 
i. What process was involved? 
ii. What were the opportunities? 
iii. What were the challenges? 


Thank you for your time and thoughtful responses. 
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Appendix D. Summary of supporting analysis documents 


Supporting analysis documents include profiles of the tools used in the eight states selected for interviews (table 
D1); images of a rubric and a rating scale (figures D1 and D2); examples of tool types by level of implementation 
(figures D3—-D6); a description of what tools are designed to address, if they request respondents to provide 
evidence to justify ratings, and how these vary by tool type (table D2); and the proportion of key multi-tiered 
system of supports/response to intervention (MTSS/RTI) practices addressed by each tool type (table D3). 


Table D1. Profiles of tools used in the eight states selected for interviews 


Florida 


Profile 


Tool details 


Name 
Description 


Intended 
administrators 


Recommended 
frequency of 
administration 
Intended users and 
recipients of data 


Rationale for 
development 
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Self-Assessment of MTSS Implementation (SAM Version 2.0, October 2015) 

The SAM is a rubric intended to assess current implementation levels of a multi-tiered system of 
supports (MTSS) model. The Florida Problem Solving/Response to Intervention (PS/RtI) and Positive 
Behavior Intervention and Supports (PBIS) projects developed the tool in 2013 and modified it in 
2015. 

The SAM is intended to be administered at the school level in grades K-12 and addresses academic, 
behavior, and social-emotional supports. 

The SAM also includes items that assess processes and activities that facilitate MTSS implementation 
(for example, leadership and capacity building). 

School-Based Leadership Teams (SBLT) complete the SAM. SBLTs are composed of six to eight staff 
members who have a leadership role in facilitating MTSS implementation at their school and should 
include representatives from administration, general education, special education, student services, 
and content specialists. 

SBLT members individually complete the SAM, and then a facilitator supports the team in reaching 
consensus on a score for each item. 

The SAM is intended to be administered once a year. 


The SBLTs review the the school-level results. District-Based Leadership Teams (DBLTs) review district- 
level results and the results of individual schools. SBLT and DBLT members may find it helpful to work 
with a coach or facilitator who can support data interpretation and are encouraged to share the data 
with instructional staff members. 

The Florida PS/RtI project staff receive the data for schools they work with for analysis and reporting. 
SBLTs should review their data from the SAM to determine strengths and steps needed to build their 
MTSS system. The Florida PS/RtI project provides a template schools may use to guide them through 
this process. The goal is for schools to use the data and tool as part of their school improvement 
process and to support action planning. 

Educators had difficulty understanding how to complete multiple implementation assessments across 
the different content areas (RTI, PBIS, and mental health), and there was some pressure to integrate 
the tools and systems. The Florida PS/Rtl and PBIS projects also wanted to be proactive in developing 
a tool that districts and schools would have confidence in if the tool were ever to be compulsory. 
Because of the movement toward integrating tools and systems and the potential for use of the tool 
to become compulsory, and because no existing tool looked at the system in an integrated way, the 
PS/RtI and PBIS projects developed a new tool. 


Florida 


Tool details Profile 
Development e The tool was co-developed by a five-member team from the Florida PS/RtI project and the state’s PBIS 
approach project, which are both funded through federal special education flow-through dollars. The team 


solicited feedback from other staff, including the leaders and staff who provide direct technical 
assistance and training. 

e The team reviewed the literature and consulted with colleagues to determine domains and indicators. 
An expert review panel of 11 district-, state-, and national-level experts on MTSS and PBIS 
implementation reviewed the draft tool and provided feedback using a structured process of rating 
items for relevance and clarity. Using qualitative feedback from reviewers, the team revised items for 
which the expert panel lacked agreement on relevance and clarity. 

e The team conducted cognitive interviews with typical representatives from SBLTs to assess the extent 
to which the target audience interpreted the items consistent with their intended meaning. Team 
members took notes and used the information to clarify terms or items. 

e The projects conducted small- and large-scale pilots to collect information about technical adequacy, 
usability, and feasibility. Florida conducted a small-scale pilot (100 schools) to inform revisions and a 
large-scale pilot (400-500 schools) to look at the properties of the instrument. 


Training and e Schools may volunteer to work with the PS/Rtl project team, but doing so is not required. Regional 
coaching on how project coordinators oversee administration of training to individuals for facilitating SAM completion. 
to use the tool The team decided to train facilitators because it is a team-based assessment and having a facilitator 


can help guide the consensus-building process. 

e Support is tailored to each district and school. Some districts have built their own internal capacity for 
training, which is a goal of the PS/RtI project. 

e Each year, the PS/Rtl project team sends districts they have worked with a link to an online database 
in which school teams enter their ratings. Project staff work with district contacts to monitor 
completion of the tool and to make school-level charts and action plan templates to help schools with 
decisionmaking. 


Challenges, e The PS/RtI project acknowledged that, like Florida, projects or states developing tools may wrestle 
recommendations, with the tension of fidelity and variability and how to determine acceptable levels of both. The 
and lessons approach to this challenge will determine the type of tool to be used, how to communicate the 


purpose of the tool, and what trainings to offer. 

e Because of the complexity of MTSS, it can be challenging for districts and schools to figure out where 
to start in using the SAM data. In some cases, it can take years to improve on a single component in 
MTSS. If MTSS is not part of a larger system, it can be difficult for districts to make changes. 

e States may have difficulty developing an integrated tool that includes both academic and behavioral 
interventions. It is easy conceptually but can be difficult technically when perspectives and expertise 
vary. The benefits of having staff from the RTI and PBIS state-funded technical assistance projects 
working together, however, outweighed the difficulty, with the projects noting that the collaboration 
resulted in common terminology and paved the way for similar collaboration at the district level. 

e The Florida projects recommend embedding the tool as part of the overall approach of 
implementation, making it clear how data from the tool support implementation and providing 
districts with coaching or technical assistance to use the data for decisionmaking. 
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Name 


Description 


Intended 
administrators 


Recommended 
frequency of 
administration 
Intended users and 
recipients of data 
Use 


Rationale for 
development 
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Kansas 


Profile 


Checklist for Implementation Readiness (2015) 

Process Implementation Tool (2017) 

Integrated MTSS Implementation Scale—Administrators (2017) 

Integrated MTSS Implementation Scale—Instructional Staff (2017) 

The Checklist for Implementation Readiness and the Process Implementation Tool (also referred to as 
the “checklists”) examine whether structures and decisions are in place to promote successful 
implementation of a multi-tiered system of supports (MTSS) in grades preK—12. Kansas’s Innovation 
Configuration Matrix (ICM) describes the principles and practices of the state’s MTSS framework 
(Kansas MTSS and Alignment) and is intended to be used as a companion document with the state’s 
implementation checklists. 

The Integrated MTSS Implementation Scales (also referred to as “implementation surveys”) are self- 
assessments of individual implementation and perceptions of school and district implementation of 
Kansas MTSS and Alignment. Data are verified by MTSS state trainers through facilitated District 
Implementation Team data analysis meetings. 

The checklists and original implementation survey (the School Implementation Scale) were developed 
in 2009, with revisions to the checklists in 2015; revisions to the implementation survey in 2015 and 
2016, creating separate items for administrators and instructional staff; and new implementation 
surveys, adapted from the revised 2016 surveys, developed in 2017. 

District and school leadership teams are trained in administering and using the data from these tools. 
In some smaller districts, all staff may attend the Kansas MTSS and Alignment trainings and be trained 
in using these tools, regardless of their leadership team status. 

The checklists are used in an ongoing manner for districts working through the process. 

The implementation scales are administered once a year. 


Administrators, instructional staff, and Kansas MTSS and Alignment state staff are expected to use the 
data generated by the tool. 

The Checklist for Implementation Readiness is a readiness-for-implementation assessment, a way to 
monitor implementation of Kansas MTSS and Alignment components and identify additional training 
and support needs. 

The Process Implementation Tool is used to ensure that structures are in place and are being used as 
intended. 

The implementation scales are used to support decisionmaking at the school and district levels and to 
inform customized coaching. 

Kansas developed the Checklist for Implementation Readiness and the Process Implementation Tool 
to help schools and districts put in place structures and processes for sustained implementation. 
Kansas developed the Process Implementation Tool to help schools and districts attain fidelity to 
processes and practices, including data-based decisionmaking. 

Kansas adapted the implementation scales to support consistent implementation across districts and 
understanding and implementation of key Kansas MTSS and Alignment elements. 
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Kansas developed and adapted its tools using funding from Grant Title VI Part B of the Individuals with 
Disabilities Education Act. A team of state educators and national experts in various content areas 
developed the ICM, which is meant to support use of the tools. 

Kansas adapted the implementation scales from an existing survey by Erickson, Noonan, and Jensen 
(2012). National experts and local educators from a university research group assisted with the 
wording and validation of the surveys. District administrators also provided feedback on the surveys. 
The Kansas MTSS and Alignment team and trainers developed the Checklist for Implementation 
Readiness and Process Implementation Tool. 

The evaluation team led pilot testing with the checklists and surveys in 2015. The checklists were 
piloted by state trainers with school districts in training. The implementation surveys were piloted 
with districts participating in implementation professional development. 

The evaluation team and the Kansas MTSS and Alignment project collaborated on revisions to the 
checklists and implementation surveys. This included adjusting, adding, and removing items on the 
checklists and modifying items on the survey scales to reduce subjectivity and promote consistency 
across respondents. Focus groups with school leadership were conducted to determine accuracy and 
usefulness of the data for decisionmaking. 

The state uses its existing training network, specifically the Kansas MTSS and Alignment project, to 
provide training on implementation assessment tools. 

Districts must apply to participate in training, which includes paperwork and onsite interviews. 
Training generally includes 12-15 days of training for the building leadership teams; ongoing in- 
district coaching during structuring, installation, and implementation; and 6-8 days of training on how 
to use the data during initial implementation. 

The tools and training around the tools are used only in schools and districts that are active in Kansas 
MTSS and Alignment implementation training or sustaining implementation. The training content is 
consistent across all schools and districts. It is tailored to the specific needs of the districts (for 
example, how to set up the online survey system to allow reporting of data at all appropriate levels). 
To build capacity in districts, the Kansas MTSS and Alignment team trains district coaches in all the 
processes, so they can be in the buildings daily to coach staff and onboard new staff. 

Schools do not use the tools on their own without guidance from a state coach for three to four years. 
District-level trainings were challenging because districts had difficulty committing the time to doing 
the professional development. This was part of the reason the application process was put in place; 
the state can assess districts’ readiness and prepare them for what will take place during training and 
implementation. 

Kansas notes that tool development takes time to make it consistent with the model, while also 
providing actionable information. 

Kansas notes that districts and schools have questions about their data and a need to look at their 
data in different ways. Users of the data can be overwhelmed by too much data, so there is a need to 
provide the information in the most concise manner possible. 
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North Carolina Self-Assessment of MTSS (NC SAM) (2015) 

The NC SAM is an adapted version of Florida’s Self-Assessment of MTSS (FL SAM). 

This rubric is intended to be an implementation fidelity tool. 

The tool is administered in grades preK—12 at the school level for academics, behavior, and social- 
emotional development. 

The district MTSS (multi-tiered system of supports) coordinator or member of the MTSS District team 
facilitates the administration of the tool with a school team. Each member of the school team reviews 
the items on the NC SAM independently before team members discuss their scores together to reach 
agreement on what answer best represents the current status of implementation at their school. 
The school team proposes a score and provides evidence, which is reviewed by the district facilitator. 
North Carolina decided to require the use of an external facilitator to increase the tool’s validity. 

The NC SAM is administered once a year during April through June. 


The school team and the district use the data to identify and prioritize next steps in implementation. 
The state receives the data when schools and districts use the online application to score their 
assessments. 

The NC SAM is used for continuous improvement at the school and district levels. 

The state also uses the data for evaluation, with the goal of developing its own universal screening 
and progress monitoring tools and eventually linking implementation data to student outcome data. 
North Carolina uses a web-based program to generate scores from the tool that are then shared with 
the schools and the state. The NC SAM produces two scores for each domain: a percentage of 
implementation and an implementation level (not implementing, initially implementing, or fully 
implementing). 

North Carolina tied items from the NC SAM to its school improvement planning software to help it use 
the NC SAM data to plan for school improvement. 

North Carolina adapted the FL SAM to better align it with its needs. North Carolina wanted to move it 
from a self-assessment to a facilitated tool; add suggested supporting evidence for every item to 
make the tool more robust; add score levels; and divide content areas into separate items (such as 
academics, behavior, social-emotional), which were combined in single items in the FLSAM. 

North Carolina conducted a standard-setting project for use of the adapted FL SAM in the state. 
Education professionals, implementers of the NC SAM, and school education agency personnel 
reviewed the tool to ensure consistency of language with North Carolina’s MTSS model and to define 
implementation levels. 

North Carolina developed fidelity thresholds by asking stakeholders in the validation project to 
identify critical items (based in current or future policy). The stakeholders determined what scores 
must be received to be considered initially or fully implementing. Other adaptations include the 
addition of supporting evidence for each item and the requirement that the tool be facilitated by an 
outside person from the district’s central office. 

There was no formal pilot testing, but North Carolina solicited feedback from the field. On the basis of 
feedback, the state is developing its own technical manual. A goal of the state is to integrate the NC 
SAM with implementation of the Tiered Fidelity Inventory (TFl) for Positive Behavioral Interventions 
and Supports (PBIS) and to release the manual once the two tools are combined and validated. 
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The NC SAM was introduced in 2015 and is intended to be fully implemented by 2020. North Carolina 
introduced the tool using a cohort model for professional development in which local education 
agencies are grouped and training is staggered. The first cohort was selected according to the results 
of a 2014 survey assessing readiness to implement initiatives based on implementation science. Many 
of the districts in the first cohort already had some key components in place and other characteristics 
that made them ideal candidates, such as PBIS implementation fidelity and lower-than-state-average 
number of students with disabilities. Cohort 2 was a mixture of most ready and least ready; cohorts 3 
and 4 were the remaining school districts; and cohort 5 consisted of charter schools. 

District MTSS coordinators (who may also be directors of curriculum and instruction, assistant 
superintendents, or special education directors) receive the training. The tool was initially rolled out 
to coordinators through face-to-face training, online videos, technology supports, guidance 
documents, and individual modeling with some sites. The state follows up with districts, including 
through meetings with coordinators and annual retraining on the instrument. 


North Carolina notes that the challenges were associated not with selecting or adapting the tool but 


recommendations, 
and lessons 


with implementing it. The district MTSS coordinators are trained, but the transfer of knowledge from 
district to school was difficult. 

One challenge reported in the field is having to administer two different assessments—the NC SAM 
and the TFI for PBIS Implementation. The finding that completing two tools has been a burden on the 


schools has prompted the state to work on combining the NC SAM with the TFI for PBIS. 


REL 2020-017 


D-6 


Tool details 
Name 


Description 


Intended 
administrators 


Recommended 
frequency of 
administration 
Intended users and 
recipients of data 


Use 


Rationale for 
development 


REL 2020-017 


Oregon 


Profile 


Oregon Response to Instruction and Intervention (ORTIi) District Implementation Evaluation Tool- 
School Based form (DIET-SB2) is the primary tool for assessing response to intervention (RTI) 
implementation (2015). 

Oregon also uses the following supplementary documents to support RTI implementation: 

-  ORTIi District Implementation Evaluation Tool—District Based (DIET-DB2), which is adapted from 
the District Capacity Assessment from the National Implementation Research Network. It is not 
necessarily specific to RTI or multi-tiered system of supports (MTSS) implementation but is used 
to assess district capacity for any initiative or innovation. 

-  ORTIi Installation Matrix, used as a coaching tool, is a descriptive document that outlines the 
steps necessary for effective implementation. It is used by ORTIi coaches with district leadership 
teams to qualitatively track progress implementing essential RT| components over time. 
However, it is not considered a formal, quantitative assessment tool because users do not 
provide ratings or scores. 

The DIET-SB2 is used to assess the extent of implementation. It includes three parts: 

° A 30-60 minute interview/questionnaire with the school principal and literacy specialist. 

° — Areview of evidence, including the RTI manual handbook, student data systems, and completed 

meeting agendas. 

An anonymous 5-10 minute online staff survey. 

The DIET-SB2 was initiated in 2010, with the current version (2) released in 2015. It is intended to be 

used with elementary schools in the area of literacy. 


fe} 


An implementation coach from the ORTIi Center completes the tool. The evidence review portion is 
completed either during the onsite visit or following the visit. 

Trained district personnel may complete the tool as a self-assessment if the district has had three or 
more years of support from ORTIi and has received training in administering the tool. 

The tool is recommended to be administered once a year, in the spring. 


The Building Leadership Teams and District Leadership Teams are the intended users of the data. 
ORTIi Center coaches also have access to the results. 

Data from the tool are accessible using ORTIi’s Assessment System for Implementation Support data 
dashboard, which provides data visualizations for the essential components. The dashboard provides 
a district’s or school’s percentage on a given dimension, allows the user to compare data across 
schools, discover strengths and areas needing improvement, and learn about resources for items 
needing improvement. 

The DIET-SB2 tool is intended to monitor the extent of implementation and fidelity and to make 
decisions on implementation and resource allocation. 

The tool is intended to be used with the ORTIi Installation Matrix, a coaching tool to explain what 
implementation should look like at different stages of implementation. 

ORTIi coaches meet with District and Building Leadership Teams in the spring of each year and use 
data from the DIET-SB2, along with student outcome data, to evaluate implementation efforts and 
create action plans. 

When ORTIi first developed the tool, there were no available tools that aligned with the state’s RTI 
system. ORTIi wanted to be proactive in enhancing implementation of its RTI framework and wanted 
a tool that was linked to its implementation components and that could provide actionable feedback 
to the schools and districts implementing RTI. 
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The ORTIi Center developed the DIET-SB2. The center is funded by the Oregon Department of 
Education. Funding for the project comes primarily through the Individuals with Disabilities Education 
Act. 

ORTIi Center staff conducted most of the work to develop the tool, though they did solicit informal 
feedback from the University of Oregon’s Center for Teaching and Learning and Education and 
Community Supports. 

The DIET-SB2’s staff survey component was adapted from the staff survey used by the Oregon 
Department of Education with its State Personnel Development Grant schools (Erickson et al., 2012). 
The center consulted with the authors on the adaptation of the survey. 

The ORTIi Center conducted a pilot with the DIET-SB2 in 2013/14 with six districts to obtain feedback 
from users (including coaches) on content, experience, and utility of information gathered. As a result, 
center staff deleted some items, adjusted the scoring rubric and components, and rearranged the 
grouping of items. Center staff also adjusted specific components on the instrument. 


Districts apply to be part of the ORTli initiative, and participants are trained to use the tools. Districts 


coaching on how 
to use the tool 


Challenges, e 


that are not working with ORTIi can technically use the tool but might have difficulty because there 
are no additional training modules available. 

Districts working with ORTIli receive training, coaching, and technical assistance with RTI 
implementation. After the initial training and coaching sequence is complete, districts are invited to 
continue participation in trainings. 

Originally, ORTIi would work with each district for five years; now it is four years for elementary 
literacy, three years for middle school literacy, and two years for elementary math. Districts are 
required to complete the elementary literacy sequence before the middle school literacy or 
elementary math sequence. 


Oregon identified issues with capacity for training. Although the DIET-SB2 can be used independently, 


recommendations, 
and lessons 


there is lack of training and capacity for districts that want to use it on their own. Once districts are 


out of the project, ORTIi no longer has the capacity to provide direct coaching or technical assistance. 
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Using Response to Intervention (Rtl) for Specific Learning Disability (SLD) Determination: School 

Building Application (K-12) for Approval (2017) 

Pennsylvania also uses the following supplementary documents: 

° Using Response to Intervention for SLD Determination: Renewal Re-Approval Application. 
Schools using assurances that have already been approved can complete this abbreviated form 
of the application. 

° — MTSS Fidelity of Implementation Tool—Enhancing RTI. This is a companion document that schools 
may use as a guide when evaluating their current evidence relative to what Pennsylvania 
considers to be evidence of fidelity of MTSS implementation and sustainability prior to using RTI 
for SLD determination. 

The tool is a 20-item application for using RTI for SLD determination in reading, writing, and math. 

Schools submit an application, which includes evidence for each item, to the Bureau of Special 

Education (BSE). The BSE reviews the application to determine approval. 

Schools use a shorter version of the tool when seeking reapproval for using RTI for SLD determination. 

Building teams complete the tool and are encouraged to partner with regional consultants during the 

completion process. Intermediate Unit consultants come onsite to review the application prior to 

submission. 

The following building team members are required to complete the tool: superintendent, building 

administrator, director of special education/supervisor, and school psychologist. 

Schools completing the tool as their application to use RTI for SLD determination complete the full 

tool once at the beginning of the school year (in September). 

Schools complete a shortened version of the tool once every five years to renew their approval to use 

RTI for SLD determination, though the expectation is that schools will continue to use the tool once 

approved. 

BSE reviews the evidence that schools submit with the tool as part of the application. They provide 

recommendations for training and technical assistance in specific areas based on their review. 

The tool’s purpose is multifaceted. Its primary purpose is as the application for a school to use RTI for 

SLD determination. It is expected that the tool will continue to be used after approval. 

Schools may also use the tool as a needs assessment or a fidelity of implementation assessment. 

Schools can use an optional companion tool, MTSS Fidelity of Implementation Tool: Enhancing RTI, to 

help them determine whether the evidence gathered for the application meets Pennsylvania’s 

standard of fidelity of MTSS implementation and sustainability. 

Pennsylvania developed the tool after the 2004 reauthorization of the Individuals with Disabilities 

Education Act stating that states cannot require schools to use the ability achievement discrepancy to 

identify children with specific learning disabilities. 

At the time of development, no tools existed that met Pennsylvania’s needs; existing measures were 

too broad for identifying schools that may use RTI for special education identification. 
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Pennsylvania has revised the application several times since its inception in 2010, with the most 
recent version released in 2017. A three-person team developed the original application. 
Intermediate Unit consultants and representatives from the Pennsylvania Training and Technical 
Assistance Network (PaTTAN) also supported the tool’s development. Early revisions included 
requiring schools to provide evidence that elements of the tool are in place. Evidence can be physical 
evidence or narrative text. 

University consultants and existing resources, particularly those with a focus on research and 
screening, informed the tool’s development. Pennsylvania also solicited feedback from cross- 
disciplinary practitioners, including administrators, to further refine the tool. 

Pennsylvania informally piloted the initial tool with cross-disciplinary practitioners. Selection of sites 
for feedback was based on a survey indicating that the sites were implementing MTSS. Feedback from 
the field was that the tool, which had 42 items, was too long and resulted in long applications 
(approximately 250 pages). Reviewers identified items that were redundant and could be removed. 
The tool was ultimately shortened to its current 20-item version. 

MTSS consultants working at the Intermediate Units are trained on the use of the tool. Schools can 
learn about the tool through sessions hosted by PaTTAN or their local Intermediate Units. Schools 
interested in using the tool can find support through their Intermediate Units at any time. 

MTSS consultants conduct professional development and site visits to support schools through the 
application process. If a school’s application is not approved, the BSE will recommend professional 
development and technical assistance that aligns with the school’s needs. 

Pennsylvania reported a general challenge in getting educators to move toward using an MTSS 
framework. The difficulty lies in how to create a tool that both motivates people to move in the 
direction of MTSS while also serving as a fidelity of implementation tool. With regard to the tool itself, 
it is still seen by some as cumbersome, even in its current, pared form. 

Pennsylvania recommends that other states rely on best practices around MTSS (for example, 
screening, data-based decisionmaking, content-based research) for developing and implementing an 
MTSS tool and for tools to require evidence. 

The state reported that the list of schools approved by the BSE is small, and no kindergarten through 
grade 12 schools have been approved. 
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Vermont Multi-tiered System of Supports Response to Intervention and Instruction (MTSS-RtIl) Self- 
Assessment 1.0 (SAT 1.0) (2014) 

Vermont released the SAT1.0 in 2014. 

The SAT1.0 was intended to support Supervisory Unions/Districts (SU/SD) as a readiness tool and 
implementation check of the MTSS Field Guide 1.0 (FG1.0). 

SU/SD leadership teams sometimes used this tool with schools within its organization to assess where 
each school stands in regard to the larger system. 

Vermont used a Data2Action tool aligned with FG1.0. This tool uses a plan-do-study-act improvement 
science cycle approach to support educators in using data to inform and improve practice (2014). 

The SAT1.0 is designed to be completed collaboratively by SU/SD/school leadership teams. 

Required members are administrators, classroom teachers, special educators, reading and math 
specialists, and, when appropriate, curriculum specialists, assessment coordinators, school 
counselors, and behavior specialists or positive behavioral interventions and supports (PBIS) 
coordinators. 

At least one administrator should be knowledgeable about FG1.0 to facilitate the SAT1.0 process. 

The SAT1.0 is recommended to be administered annually by SU/SD/school leadership teams. 


SU/SD/school leadership teams are the users of this tool. 

The data are shared with administration and staff to inform the Comprehensive Needs Assessment of 
their Continuous Improvement Plan (CIP). 

The purpose of this tool is to support SU/SD/school leadership teams to adopt, develop, or maintain a 
multi-tiered system of supports (MTSS) designed to improve instruction and intervention for all 
students in the areas of behavior and social-emotional and academic learning. 

The SAT1.0 is used to support leadership teams in reflecting on each component of the Field Guide to 
identify areas of strength and need. 

Teams identify initiatives in place that can support implementation, prioritize areas, identify concerns 
to be addressed in action planning, and determine next steps. 

Teams use an iterative process to perform these tasks. 

The SAT1.0 was developed to support supervisory unions, districts, and schools in determining 
readiness and fidelity of implementation of the FG1.0. 

The SAT1.0 was created to reflect the five components of an MTSS in Vermont. 

The SAT1.0 contains a list of characteristics for each of the five components of the FG1.0 for 
SU/SD/school leadership teams to judge where their SU/SD/school stands on each characteristic. 

In 2012 Vermont contracted with the Vermont Reads Institute (VRI) to create the FG1.0 and SAT1.0. 
VRI collaborated with the Vermont Statewide Steering Committee for RTI, which included 
stakeholders from the Vermont Agency of Education; educators from elementary, middle, and high 
schools; and consultants on guidelines development. 

Vermont also used a consultant with background in implementation science, as well as external 
systems coaches, to help schools with their tools. 

The State Personnel Development Grant (SPDG) team contracted with a consultant working with 
Michigan’s Integrated Behavior and Learning Support Initiative. 

Vermont piloted the FG1.0 and SAT1.0 with SPDG schools. External system coaches used the pilot 
results as part of the evaluation requirements of the SPDG. 

The SAT1.0 was not tested technically for validity or reliability metrics. 
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Training and e When the SAT1.0 was initially developed, Vermont provided regional presentations on the tools and 
coaching on how FG1.0. 

to use the tool e The state launched a pilot program in 2014, which was funded using SPDG money. 


e External systems coaches were provided only to the SPDG schools, though MTSS training was 
available to all schools in the state. 

e During the pilot, coaches facilitated conversations on use of the FG1.0 and SAT1.0. 

e Vermont followed up with schools participating in the SPDG grant during the grant period. However, 
now that the grant has ended, there is no longer any follow-up for trained schools. 

e Vermont offered MTSS implementation training as part of a week-long summer institute. 

Challenges, e Vermont experienced funding and capacity constraints that limited what was offered to schools. 
recommendations, e After the SPDG grant ended in 2017, Vermont was no longer able to fund external systems coaches 
and lessons used during the SPDG. 

e Vermont created a dedicated team to support implementation of MTSS (2016). 

e Vermont recognized the need to revise its FG and SAT since its adoption in 2014. 

e Through annual MTSS survey data, schools indicated a continuing need for supports in their systemic 
and comprehensive approach and in high-quality instruction and intervention, two of the five 
components of Vermont’s MTSS framework. 

e Vermont contracted with a vendor to create FG2.0 and SAT2.0. 

e Vermont learned that it is imperative to show connections between the FG2.0 and other state 
documents used to support SU/SD/school leadership teams (for example, CIP). 

e Beginning in 2019 the Vermont MTSS team started to provide technical assistance and professional 
learning to support the use of these revised tools. 
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Schoolwide Implementation Review (SIR) (2015) 

Rtl All Staff Perception Survey (All Staff) (2016) 

The SIR is a self-assessment rating scale intended to assess implementation of Wisconsin’s response 
to intervention (Rtl) framework in schools. The SIR was developed in 2011 and updated in 2015. 
The All Staff is a supplemental survey that assesses staff ratings on infrastructure, initial 
implementation, or full implementation against the ideal Rtl practice in the school and can be 
compared to the leadership team’s ratings on the SIR. 

The tools are designed to be administered at the school level (grade levels not specified) and are 
intended to be completed from a reading or math perspective. 

The leadership team, which completes the SIR, includes at least three people, one of whom is a 
building administrator and the others are a cross-section of staff (for example, principal, general 
educators, special educators, student services staff, and content or service experts). 

The leadership team determines who receives the All Staff survey. The Wisconsin Rtl Center’s 
guidance is that all adults who work with students in the building take the All Staff survey. 

The SIR is recommended to be taken annually. 

The All Staff survey is recommended to be taken annually. 


All levels of the education system receive the results of the SIR. The Wisconsin Rt! Center reports to 
the Wisconsin Department of Public Instruction, in aggregate, the impact of the work of the 
Wisconsin Rtl Center and schools’ progress toward implementation of Wisconsin’s Rtl vision. Subscale 
and item-level results are used to facilitate professional development plans at the regional and state 
levels. 

Districts and schools use the data from both the SIR and the All Staff survey to plan actions for 
improvement around subscale scores or specific items from the scale to determine progress toward 
full implementation. 

Assessment results are used to determine training frequency and location and the impact of turnover. 
The Wisconsin Rtl Center also uses the results to revise and validate the instruments over time. 
Wisconsin chose to develop the SIR because no tools were available to adequately assess Rtl at the 
systems level when the state initiated its work in 2010. Districts and schools were posing questions to 
the state on how to establish an infrastructure for Rtl and how to improve their understanding of 
Wisconsin’s Rtl framework and related practices. 
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Wisconsin used federal special education discretionary grant dollars to create the Wisconsin Rtl 
Center, which was tasked with ensuring Rt! implementation and creating tools to support Rtl 
implementation. 
To develop the tools, the Wisconsin Rtl Center drew on expertise from the National Implementation 
Research Network, the national Positive Behavioral Interventions and Supports (PBIS) Technical 
Assistance Center, and American Institutes for Research consultants from the National Center on 
Response to Intervention. 
The All Staff survey was modeled after a tool from the PBIS National Technical Assistance Center- 
Benchmarks of Quality and the Schoolwide Assessment Survey. 
The Wisconsin Rtl Center also received assistance from a doctoral candidate and an evaluation 
coordinator to examine technical properties of the SIR, which has been reported to have high internal 
consistency, internal validity, and convergent validity. 
The Wisconsin Rtl Center’s Regional Technical Assistance Coordinators facilitated the piloting of the 
tools. The SIR was piloted in 2011 with 24 schools—11 schools engaged in focus groups on the SIR 
(spring 2011), and 13 schools completed the SIR and provided feedback in an Excel form (fall 2011). 
Wisconsin made updates to the tool in fall 2011, including changes to language and scoring, before it 
was placed online for use. On the basis of feedback during the pilot, the Wisconsin Rt! Center also 
created an FAQ document and a webinar on how to take the SIR. Wisconsin piloted the All Staff 
survey with a subset of the schools that received the SIR. 
The state recommends that school teams take the SIR as part of the Wisconsin Rtl Center’s 
Framework Training. The Center offers two options for helping schools develop background 
knowledge to take the SIR: 
° A two-day Wisconsin Rtl Center foundational training held throughout the school year and 
across the state (recommended). The state trains its academic Regional Technical Assistance 
Coordinators to train school teams on the SIR, and Cooperative Educational Service Agency 
consultants to support school teams in taking the SIR. This mode of training was selected to 
standardize trainings and create fidelity to the model. 
o Alternatively, schools can participate in an online module about completing the SIR, available on 
the Wisconsin Rtl Center website (recommended as a refresher or for new team members). 
School trainings are advertised through the Wisconsin Rtl Center website, newsletters, and social 
media and are held regionally about five times a year with 50-60 people at each training. 
The Wisconsin Rtl Center continues to offer support for schools following trainings through refresher 
and booster trainings, networking meetings, communities of practice, virtual and onsite technical 
assistance, email support, and periodic reminders to take the SIR. 
Wisconsin Rtl Center uses a recognition program that encourages tool use because the state does not 
require it. Schools are recognized for their implementation progress at conferences, through press 
releases, as an example in the annual report, and with seals on emails. 
Wisconsin found it challenging to define and assess Rtl at the same time. Because of the concurrent 
introduction of Rtl to the state and the roll-out of the SIR, schools were learning about Rtl at the same 
time they were learning how to assess its implementation. 
Wisconsin found it difficult to develop the tool when there were few examples and lessons to be 
learned from other states that had engaged in a similar development process. Similarly, the state 
found it difficult to develop the tool when its definition of Rtl was broader than that in the national 
conversation at the time. 
The state suggests that, given the wide availability of tools today, states consider adopting or 
adapting existing tools before developing their own. Wisconsin suggests that, when implementing a 
tool, a state should be clear on how data will be used and maintained and should know its logic model 
or end goal. 
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approach 
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MTSS Implementation Checklist for Literacy (Literacy Checklist; 2016) 

Wyoming also uses the National Center on Response to Intervention’s Essential Components of RTI 
Integrity Rubric and Worksheet (NCRTI Rubric) without modification (2014). However, the state 
changed the name of the tool to be the Wyoming MTSS Fidelity of Implementation Rubric 
(Implementation Rubric). 

The Literacy Checklist assesses the extent to which school personnel are implementing the core 
components of multi-tiered system of supports (MTSS) in the topic area of literacy. 

It is used in conjunction with the Implementation Rubric. 

Districts participating in the onsite training through the State Personnel Development Grant (SPDG) 
are intended to administer the Literacy Checklist and the Implementation Rubric. Non-SPDG districts 
also have an opportunity to participate in trainings and administer the tools. 

The state coach recommends that the Implementation Rubric and Literacy Checklist be administered 
once a year. 


Districts enter their information electronically and submit it to statisticians for the SPDG evaluation. 
MTSS coaches have access to the information online prior to submission to statisticians. 

The Implementation Rubric is used to help schools assess whether they are implementing the 
essential MTSS components and where they need improvement. 

After the Implementation Rubric is administered, the Literacy Checklist is used to determine which 
school staff are implementing the core features of MTSS for literacy and to help schools develop next 
steps for planning. 

Wyoming developed the Literacy Checklist as a follow-up to the Implementation Rubric to gain more 
specificity around literacy for some of the items used by school teams. Literacy was an area in which 
Wyoming thought it could make a difference, and it wanted to ensure that research from the National 
Reading Panel report was included in the tool. 

Tool development was funded by Wyoming’s SPDG project, the Wyoming Instructional Network— 
Improving Outcomes for Students with Disabilities (Project WIN), which was funded from 2011 to 
2016. A no-cost extension has allowed Project WIN to continue funding to support districts with MTSS 
implementation, including training on the use of the tools. 

The state Department of Education, MTSS state coaches, and experts from the NCRTI assisted with 
the development of the Literacy Checklist. The NCRTI experts assisted with aligning the Literacy 
Checklist with the Implementation Rubric. Team members participating in the development of the 
Literacy Checklist brought expertise in coaching, implementation, MTSS, and statistics. 

The tool was influenced by Montana’s tool and the literature on reading and development. 

The Literacy Checklist was published in 2016 and is currently undergoing informal field testing. The 
state is asking districts and schools about what is and what is not working when they use the tool. 
They are learning that the tool is too long and have plans to reduce it. 

There is no information about the technical properties for either the Literacy Checklist or the 
Implementation Rubric. 
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Tool details 
Training and 
coaching on how 
to use the tool 


Challenges, 


Wyoming 


Profile 


Teams of two to five members from each school are trained onsite by an MTSS coach. Trained team 
members include administrators, teachers, service providers, and school psychologists. Districts that 
have not been part of the trainings may also receive technical assistance from the MTSS coach. 

A single MTSS coach conducts onsite trainings. Training is supported by funding from the SPDG. 
Schools using SPDG money are required to be trained on the state’s MTSS tools, though non-SPDG 
schools may volunteer to be trained. Approximately 8-10 school districts (of a total of about 48) in 
Wyoming are actively engaged in MTSS implementation using both tools. 

Training consists of four days of onsite training on seven modules over the course of a year. There are 
also two coaching conference calls a month. The onsite training is the same across schools. When 
working with individual teams, the coach will tailor the coaching to the school’s specific needs. 
Districts may request additional follow-up after training. 


Wyoming notes that developing a tool is time intensive and may not be necessary, given that there 


recommendations, 
and lessons 


are many tools available today that are similar to its tool. During tool development, there was often 
difficulty reaching consensus on items because of the varied perspectives and expertise of those 
providing input. 

The data from the tools are helpful in planning next steps, but the length of the Literacy Checklist can 
be problematic. The alignment of the Literacy Checklist with the Implementation Rubric has led to 
questioning of the overlap and of the need for two tools. Wyoming continues to look for ways to 
reduce the length of the tools and the implementation burden on districts and schools. The state 
recommends selecting only a few items to focus on at a time. 

Wyoming also notes that one challenge is a lack of directives and clear guidelines at a state level on 
MTSS and its implementation. For example, there are no formal guidelines or protocols for following 
up with schools after training. The state also funds several optional initiatives, including MTSS, and it 
is not clear to schools how they are all aligned and in which initiatives they should be participating. 


Wyoming is also interested in embedding more implementation science within its tools. 


Images of a rubric and rating scale 


Figure D1 provides a screenshot of a rubric (North Carolina’s Self-Assessment of MTSS), and figure D2 provides a 
screenshot of a rating scale (Wisconsin’s School-Wide Implementation Review). 
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Figure D1. Screenshot of a rubric tool: North Carolina’s Self-Assessment of MTSS 


~~ MULTI-TIERED SYSTEM OF SUPPORT 


NC Napartmant of Public Inatructian 


10: Coaching is used to support MTSS implementation 


© Coaching logs/documentation of coaching 
activities/opportunities 

* School improvement plan includes information about coaching 
supports and structures around MTSS 

© PLC/Grade Level/Department Team meetings logs evidencing 
coaching opportunities 


11: Schedules provide adequate time for trainings and 
coaching support 


AND data on professional 
development, 
implementation fidelity, 
and student outcomes are 
used to refine coaching 
activities 


No coaching is provided Initial " AND coaching activities are 
coaching is occurring ri 
porn ne pode thats focused prieerliy on Pencorcnoea = cece 
jementsof MTss_ | ‘@tiltating or modeling the | POT rative and 
= components of MTSS perfo a eit 


AND schedules permit 


includ: personnel to access 
AN schetiiles additional training and 


A hedule t ied t Schedules include time : 
PL ide s¢ — 2 development and allocated for trainings tne for ing coaching support that is 
* PLC/Grade level/Department agendas evidence coaching support 
coaching for MTSS differentiated based on 
support/coaching opportunities heir is 
© PD calendar 


Schedules do NOT 
include time allocated to 
professional 


AND schedules include AND schedules permit 


Schedules do NOT 


12: Schedules provide adequate time to administer 
academic, behavior and social-emotional 
assessments” needed to make data-based decisions 


© Master schedule or master calendar with time for data 


collection included 
© Assessment calendar 


13: Schedules provide adequate time for multiple tiers 
of evidence-based instruction and intervention to 
occur 


* Master schedule with evidence of intervention/instruction time 
based on needs of school population (adequate time for Core, 
Supplemental and Intensive) 


14: Schedules provide adequate time for staff to 
engage in collaborative, data-based problem-solving 


and decision-making 


© Master schedule with evidence of data-based problem-solving 


time reserved 


include time allocated to 
administering 
assessments needed to 
make decisions across 
tiers 


The master schedule is 
developed without 
student data and does 
not include time for 
multi-tiered interventions 


The master schedule 
does not provide 
opportunities for 

collaborative, data-based 


problem-solving and 
decision-making to occur 


Schedules include time for 
academic, behavior and 
social-emotional 
assessments administered 
to all students (e.g., 
universal screening) 


The master schedule is 
developed utilizing student 
data and includes time for 
multi-tiered interventions 


The master schedule 
provides opportunities to 
engage in collaborative, 
data-based problem- 
solving and decision- 
making to occur 


time to administer more 
frequent progress 
monitoring assessments to 
students receiving Tier 2 
and 3 services as specified 
{e.g., weekly or monthly 
assessments) 
AND the master schedule 
facilitates effective 
implementation of multi- 
tiered interventions 
matched to student needs 
by content area and 
intensity (Tier 1, Tier 2, Tier 
3) 


AND the master schedule 
provides sufficient time for 
the process to occur with 
fidelity 


personnel to administer 
additional assessment (e.g., 
diagnostic assessments) 
across content areas and 
tiers needed to engage in 
data-based problem- 
solving 


AND the master schedule 
allows for flexible student 
groupings 


AND the master schedule 
provides opportunities for 
collaborative, data-based 
problem-solving and 
decision-making to occur in 
settings such as: leadership 
team meetings, grade-level 
meetings, cross grade-level 
meetings, professional 
learning community 
meetings 


Adapted from The Self-Assessment of MTSS. (2013). Florida's MTSS, Florida Department of Education. Adapted with permission. 


Source: North Carolina’s Self-Assessment of MTSS (2015). Retrieved April 30, 2018, from https://www.livebinders.com/media/get/MTQzNTk4NTE=. 


Figure D2. Screenshot of a rating scale tool: Wisconsin’s School-Wide Implementation Review 


“ PART ONE: HIGH QUALITY INSTRUCTION 


High quality instruction (curriculum, instruction, and assessment) is engaging, standards-based, data-driven, and research- 
based and is grounded in culturally responsive practices. 


? 
s 
Do we have HIGH QUALITY INSTRUCTION on multiple levels? : i : 
|| 2 (28s: 
For instruction at the UNIVERSAL level, we... 


Use universal curriculum and instruction based on the Wisconsin State 
standards or local standards 


Deliver universal curriculum and instruction based on grade-level/course 
benchmarks 


Use research-based practices and/or programs within our universal curriculum and 
instruction 


4 Differentiate universal curriculum and instruction based on student needs 
5 Provide universal curriculum and instruction that engages students 


Provide universal curriculum and instruction that uses the cultural beliefs, practices, 
and experiences of our students 


Use formal strategies to share our grade-level/course benchmarks with all 
parents/guardians 


Use a process to ensure that our universal curriculum and instruction are delivered 
with fidelity (i.e. as intended) 


Use multiple measures to review the overall effectiveness of our universal 
curriculum and instruction for all students and adjust accordingly 


Use multiple measures to review the effectiveness of our universal curriculum and 
instruction for demographic groups of students and adjust accordingly 


10 


Source: Wisconsin’s School-Wide Implementation Review (2015). Retrieved April 23, 2018, from 
https://www.wisconsinrticenter.org/assets/files/SIR/PrintFriendlySIR.pdf. 


Examples of tool types by level of implementation 

The following examples illustrate how tool types (rubric, rating scale, checklist, and survey) vary in how explicitly 
they specify what is expected of schools when implementing the same multi-tiered systems of supports/response 
to intervention (MTSS/RTI) dimensions (figures D3—D6). The research team chose “Administer universal screening 
measures” and two of its dimensions (administer at least twice a year and include all students) for the examples, 
as almost all tools included these dimensions. 


In each example of a tool type, red boxes indicate the level of implementation and blue boxes indicate the explicit 
practice (dimensions). For example, the levels of implementation for the rubric in figure D3 are not implementing, 
emerging/developing, operationalizing, and optimizing. 
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Figure D3. Screenshot of a rubric on universal screening: Descriptions of expected practice for each level of 
implementation from North Carolina’s Self-Assessment of MTSS 


12: Schedules provide adequate time to administer 
academic, behavior and social-emotional 
assessments’ needed to make data-based decisions 


* Master schedule or master calendar with time for data 
collection included 
° Assessment calendar 


34: Staff understand and have access to academic, 
behavior and social-emotional data sources that 
address the following purposes of assessment: 1) 
identify students at-risk academically, socially, and/or 
emotionally, 2) determine why student is at-risk, 3) 
monitor student academic and social-emotional 
growth/progress, 4) Inform academic and social- 
emotional instructional planning, 5) determine 
student attainment of academic/behavioral outcomes 


* Assessment Plan (within or separate from MTSS implementation 
plan) 
Assessment inventory 
School Improvement plans 
Screening results and use in identifying students at-risk 
Intervention Plans 


Schedules do NOT 
include time allocated to 
administering 
assessments needed to 
make decisions across 
tiers 


Staff do not understand 
and have access to 
academic, behavior, and 
social-emotional data 
sources that address the 
purposes of assessment 


Schedules include time for 
academic, behavior and 
social-emotional 
assessments administered 
to all students (e.g., 
universal screening) 


Staff learn the purposes of 
assessment within MTSS 
and the leadership team 
selects measures for the 
purposes of assessment 

across academic, behavior 

and social-emotional areas 
that are reliable, valid and 
accessible, as well as 
culturally, linguistically, and 
developmentally 
appropriate 


time to administer more 
frequent progress 
monitoring assessments to 
students receiving Tier 2 
and 3 services as specified 
(e.g., weekly or monthly 
assessments) 

AND staff engage in 
assessment with fidelity to: 
1) answer predetermined 
guiding/critical questions 
regarding student 


nctionine/o om 


identify students who are 
at-risk at least 3-4 
times/year, 2) determine 


Ya ude re} . 
monitor student 
growth/progress, 4) inform 
instructional/intervention 
planning, 5) determine 
student attainment of 
academic, behavior, and 
social-emotional outcomes 


ND schedules perm! 
personnel to administer 
additional assessment (e.g., 
diagnostic assessments) 
across content areas and 
tiers needed to engage in 
data-based problem- 
solving 


AND the leadership team 
and/or staff collaboratively 
and systematically evaluate 

and adjust assessment 
practices to ensure 
availability of accurate and 
useful data to inform 
instruction, and assessment 
tools are evaluated for 
continued value, 
usefulness, and cultural, 
linguistic, and 
developmental 
appropriateness 


Source: North Carolina’s Self-Assessment of MTSS (2015). Retrieved April 30, 2018, from https://www.livebinders.com/media/get/MTQzNTk4NTE=. 


Figure D4. Screenshot of a rating scale on universal screening: Statement of ideal practice and the level of 
implementation from Wisconsin’s School-Wide Implementation Review 


Do we use BALANCED ASSESSMENTS to continuously review 


student progress? 


For assessment of learning at the UNIVERSAL level, we... 


times each year 


Source: Wisconsin’s School-Wide 


Implementation 


https://www.wisconsinrticenter.org/assets/files/SIR/PrintFriendlySIR.pdf. 


Review 


(2015). 


Retrieved 


Use a process toJscreen all studentsjon grade-level/course benchmarks multiple 


April 


Not in Place 
Purpose-Building 
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Figure D5. Screenshot of a checklist on universal screening: Ideal practices and the presence or absence of 
each practice from Pennsylvania’s Using Response to Intervention (Rtl) for Specific Learning Disability (SLD) 


Determination School Building Application (K-12) for Approval 


Implementation Recommendations 

and Resources: 
Fidelity of Universal Screening 
requires that a system is in place to 
assess the health of tier 1/core 
instruction for all students. Screening 
data is used to inform instructional 
practices and ultimately change 
predicted outcomes earlier rather 
than later, especially for the most 
vulnerable students. 


All students are screened a minimum 
of three times per year to assess 
whether differentiation is working to 


grow “all students” regardless of a 
student’s proficiency status. 


LEA Discussion Items: 


Indicate the instrument that you use and 
how often it is used to conduct universal 
screening each year. If the instrument 
used is NOT listed on 
www.rti4success.org, reference 
established reliability and validity of the 
screening measure and describe how it 
matches expectations for learning in a 
specific grade level. 


Describe how adherence to 
standardization and scoring rules (of test 
administration) is monitored. 


Required Documentation for Submission: 


2a. Describe the process for establishing cut 
points to identify student needs that warrant 
provision of tier 2 and/or tier 3 services. 


2b. Identify the timeline and meeting 
structure that is used to facilitate grade level 
analysis and goal setting relative to honing 
tier 1/core instruction after 
screening/benchmarking is conducted. 


Source: Using Response to Intervention (Rtl) for Specific Learning Disability (SLD) Determination School Building Application (K-12) for Approval (2017). 


Retrieved July 23, 2018, from 


https://www.pattan.net/getattachment/Multi-Tiered-System-of-Support/Response-to-Intervention-RTI/RTI-SLD- 


Determination/UsingResponselntervention_SL.PDF/?lang=en-US. 


Figure D6. Screenshot of a survey on universal screening: Ideal practices and the presence or absence of the 
practice from New York State Self-Assessment Tool for Rtl Readiness and Implementation 


To what degree is my school/district 


implementing this practice? 


ing 
ion 


not currently 
lement 
lementat 


imp 


imp 


Assessment: Screening 


1. My school/district has an already identified screening/benchmarking tool 


that will be used for determining at-risk status. 


2. My school/district has established a regular schedule fo 


students in grades K-4 a minimum offthree times per year,] 


Source: 


New York State Self-Assessment Tool for Rtl 


Readiness and 


Implementation 


(2010). 


Retrieved August 


implementation 


21, 2018, from 


https://nysrti.org/files/documents/resources/forms/new_york state self assessment tool for rti_ readiness as word doc updated 1 8 14.pdf. 
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What the tool is designed to assess 


What the tool is designed to assess and whether evidence to justify ratings is requested varies by tool type (table 
D2). 


Table D2. What the tool is designed to assess and whether evidence to justify ratings is requested for 31 tools, 
by tool type 


What the tool is designed to assess 


Number of tools assessing 


INU Taal ot=Yarey mxote) [sy stages of an 
assessing fidelity to anyet=yant=rarecialeyal NU Taa) ol=lareymxete) [sy Number of tools not 
expected practices along continuum against only an requesting evidence to requesting evidence 
Tool type a continuum ideal practice justify rating to justify rating 
Rubric 13 0 10 3 
Rating scale 0 6 4 2 
Checklist 0 4 2 2 
Survey 2 6 0 8 
Total 15 (48 percent) 16 (52 percent) 16 (52 percent) 15 (48 percent) 


Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 


Proportion of key practices addressed by each tool type 


Table D3 provides information on the tools that address at least 75 percent of the components, subcomponents, 
and dimensions and at least one of the other three key characteristics of a well-developed tool (the tool requests 
evidence, the state conducted a pilot test of the tool, or the tool has publicly available technical adequacy 
information). The average for each tool type (rubric, rating scale, checklist, and survey) shows how these 
highlighted tools compared with other tools of the same type. 
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Table D3. Percentage of key multi-tiered systems of supports/response to intervention practices addressed by each tool type, by state 


Tool has publicly 
State EWE Evel 


Components Subcomponents Dimensions 


addressed addressed addressed aye) feroyate|Urein=ve technical 


(percentage) (percentage) (percentage) requests _ pilot test adequacy 
mote) Mar-lani=s (N = 4) (N = 8) (N = 24) evidence fo} mKeJe)| Taicelaaareiucelal 


Rubric. An assessment tool that articulates the expectations for implementation of multi-tiered system of supports (MTSS)/response to intervention by providing 
coherent criteria and descriptions of performance levels for these criteria. Rubrics describe, rather than judge, levels of performance.? This tool is completed by a 
school- or district-based team. (n = 13) 


Minnesota Minnesota Reading Tiered Fidelity 100 100 75 Yes No® No 
Inventory 
All 13 rubric tools Average of all 13 rubric tools 77 82 53 n=10 n=5 n=1 


Rating scale. An assessment tool that presents a list of ideal MTSS/RTI practices with a place for observers to mark the degree to which each practice is displayed (for 
example, not in place, purpose-building, infrastructure, initial implementation, and full implementation). Rating scales do not provide descriptions for each level of 
performance but simply state the ideal practice. This tool is completed by a school- or district-based team. (n = 6) 


Wyoming‘ MTSS Implementation Checklist Literacy 100 100 83 Yes No No 
All 6 rating scale tools Average of all 6 rating scale tools 58 75 50 n=4 n=3 n=2 


Checklist. An assessment tool that presents a list of ideal MTSS/RTI practices, with a place for observers to mark whether that practice is present or absent. Checklists 
do not provide descriptions of levels of performance for each ideal practice. The tool is completed by a school- or district-based team. (n = 4) 


Pennsylvania‘ Using Response to Intervention for SLD 75 88 75 Yes No No 
Determination: School Building Application 
(K-12) 
All 4 checklist tools Average of all 4 checklist tools 56 75 31 n=2 n=2 n=0 


Survey. An assessment tool that collects information from a sample of specific individuals (for example, teachers) who report to what degree or whether ideal 
MTSS/RTI practices are present or absent in their school or used by them. The tool may include a numeric or categorical rating scale or yes/no response options. (n = 8) 


New York New York State Self-Assessment Tool for 100 100 75 na Yes No 
Rtl Readiness (targets elementary schools) 


All 8 survey tools Average of all 8 survey tools 66 78 31 na n=5 n=1 


na is not applicable. 

a. Brookhart, 2013. 

b. State personnel reported that they were piloting the tool at the time data for the study were being collected. 
c. State was part of the interview sample. 

Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 


Appendix E. Addressing academics and behavior in multi-tiered system of supports and 
response to intervention tools 

Of the 21 states in this study, 18 used at least one tool that addressed both academics and behavior, 12 used a 
multi-tiered system of supports (MTSS) framework, and 9 used a response to intervention (RTI) framework. In line 
with the definition of MTSS (see box 1 in the report), all 12 states using an MTSS framework had tools that included 
a focus on at least one academic subject or behavior (table E1). Even though RTI is often thought to focus primarily 
on academics, 6 of the 9 states that used an RTI framework had tools that addressed both academics and behavior. 
The use of RTI to describe a tiered system of supports for both academics and behavior provides evidence that 
some states may be using MTSS and RTI interchangeably. However, other states are using the term RTI to refer 
specifically to academics. 


Similarly, all 19 of the 31 tools intended to assess implementation of an MTSS framework included a focus on at 
least one academic subject area and behavior. Half the tools (6 of 12) that were intended to assess an RTI 
framework addressed both academics and behavior (table E2). 


Two interview respondents discussed the opportunities and challenges of creating a tool that merges academics 
and behavior within an integrated framework. Both respondents spoke about the benefit of having staff from the 
state’s MTSS or RTI and positive behavioral interventions and supports initiatives engage with each other to 
establish common terminology. However, one respondent remarked that while everyone agrees conceptually that 
there should be integration and alignment of academic and behavioral interventions, achieving this technically is 
complicated. Some of the challenges cited were negotiating different philosophies about the emphasis that should 
be placed on teacher referrals, coaching models, and essential topics for professional development. 


Table E1. Topical focus of tools for assessing implementation of multi-tiered system of support/response to 
intervention frameworks at the state level 


Number of 
states Percent of 


Topical focus (N= 21) states 


Multi-tiered system of supports 


General academics and behavior 8 38 
Math, literacy/reading, and behavior 2 10 
Literacy and behavior Z 10 


Subtotal 12 57 


Response to intervention 


General academics and behavior 6 28 
Literacy and math 1 5 
Literacy 1 5 
Schools seeking approval to use response to intervention for specific learning disability 1 5 
determination for reading, writing, or math 

Subtotal 9 43 


Note: Percentages do not sum to 100 percent due to rounding. 
Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 
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Table E2. Topical focus of for assessing implementation of multi-tiered system of support/response to 
intervention frameworks at the tool level 


Number of 


tools Percent of 
Ko) o)er-] Maelab hs (N = 31) inere) ks 


Multi-tiered system of support 


General academics and behavior 9 29 
Reading, math, and behavior 4 13 
Literacy and behavior 5 16 
Math and behavior 1 3 
Subtotal 19 61 
Response to intervention 

General academics and behavior 6 19 
General academics 1 3 
Literacy and math 1 3 
Literacy 2 6ss 
Math 1 3 
Schools seeking approval to use response to intervention for specific learning disability 1 3 
determination for reading, writing, or math 

Subtotal 12 39 
Total 31 100 


Note: Percentages do not sum to 100 percent due to rounding. 
Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 


